Projected-Feature Scale Calibration for 2-Bit Vision Encoders in Multimodal Large Language Models

Abstract

Very low-bit quantization of a multimodal large language model’s vision encoder can impair text recognition and chart understanding. We present projected-feature scale calibration (PSC), an image-only post-training method for 2-bit vision encoders. PSC starts from range-searched GPTQ, fixes integer codes and zero points, and jointly updates existing group scales by matching features through a frozen multimodal projector. This objective aligns image-only calibration with the visual representation supplied to the language model. The learned updates fold directly into the original scales, preserving the inference architecture. Experiments on LLaVA-OneVision show consistent improvements in text recognition and chart understanding over the same strong quantized initialization, including under simultaneous language-model compression. Matched controls support projected-feature supervision over encoder-output reconstruction, repeated calibration samples confirm stable gains, and an unchanged-recipe contrastive language–image pre-training (CLIP) evaluation supports transfer to a second encoder. PSC thus improves the accuracy of compact vision encoders while preserving their quantized representation.

1. Introduction

Multimodal large language models (MLLMs) connect visual perception with language generation, enabling a single model to answer questions about natural images, written text, and structured visual content [1,2,3]. In a common architecture, an image encoder produces patch features [4], a multimodal projector maps these features to the language representation space, and a language model generates a response [5]. Compressing the language model addresses only one part of this pipeline [6,7,8]. Weight quantization can reduce model storage [9]. In an MLLM, this also motivates compressing the vision encoder [10], whose weights are needed for each visual input. Moreover, high-resolution processing can divide an image into several crops, increasing the amount of visual computation without reducing the encoder’s weight footprint [2]. This vision-weight budget is particularly relevant when the encoder is paired with a small language backbone. Vision-encoder compression therefore complements language-model compression in efficient multimodal deployment.

The main difficulty is preserving the visual information that a downstream answer depends on. Reading a word or extracting a value from a chart can require details that are less important for broad object recognition. TextVQA, OCRBench, and ChartQA test these capabilities through complementary forms of text-centered visual understanding [11,12,13]. At 2-bit weight precision, each quantization group has only four integer levels, leaving limited freedom to represent the original weights. This makes the visual encoder a demanding compression target. Local quantization can substantially impair text recognition even after careful range selection, motivating calibration that preserves the visual features used by the language model.

Post-training quantization offers several ways to reduce this error [14,15,16]. GPTQ uses approximate second-order information to compensate for weight quantization within a layer [17]. Activation-aware weight quantization (AWQ) uses activation-aware weight scaling, while half-quadratic quantization (HQQ) provides a data-free optimization-based quantizer [18,19]. GPTAQ further addresses asymmetric calibration by accounting for differences between quantized and full-precision computation [20]. These methods establish strong starting points for low-bit models. Local reconstruction objectives [15] address errors that subsequently propagate through the visual path. Errors generated in an early visual block may be modified by later attention layers, multilayer perceptrons, and the projector. A calibration objective placed at the end of this visual path can therefore address a different aspect of the error.

Learning quantization parameters is already an established direction [21,22]. OmniQuant optimizes clipping and equivalent transformations through block-wise reconstruction [23]. More recent work includes GSQ, whose 2-bit language-model experiments add end-to-end scale-only refinement after discrete optimization, and PiSO, which studies the selection of quantization scales [24,25]. These advances motivate calibrating 2-bit vision encoders at the multimodal projection boundary, where the reconstructed features directly connect visual perception to language generation.

We propose projected-feature scale calibration (PSC), an image-only procedure that jointly calibrates vision-encoder scales through a frozen multimodal projector. PSC first constructs a range-searched GPTQ initialization and then holds its integer codes, zero points, and column order fixed. It jointly optimizes multiplicative updates to the existing scales across the vision encoder. The supervision is the difference between the projected features of the quantized student and those of the bfloat16 (BF16) teacher. Although the projector is frozen, gradients pass through it to all trainable scales. Calibration uses unlabeled images and cached visual features from the teacher. At export, the updates are absorbed into the original scale tensors.

This study makes three contributions:

• We define an image-only, projected-feature calibration procedure that refines a strong 2-bit vision quantizer while preserving its representation format;
• We demonstrate improvements in text recognition and chart understanding across LLaVA-OneVision sizes and evaluate transfer to an encoder based on contrastive language–image pre-training (CLIP);
• We examine initialization, update granularity, and supervision location, and test whether the visual gains remain under simultaneous language-model quantization.

2. Related Work

2.1. Quantization of Vision Transformers and Language Models

Vision-transformer quantization must account for the numerical behavior of attention, normalization, and feed-forward layers [26,27,28]. PTQ4ViT introduces twin uniform quantization and a Hessian-guided calibration metric, while FQ-ViT studies fully quantized vision transformers with specialized treatments of normalization and attention [29,30]. Their results show the importance of matching the quantization procedure to the visual architecture.

For large language models, GPTQ approximates a layer’s reconstruction error using calibration activations and compensates for the errors introduced during sequential quantization [17,31]. AWQ protects salient weight channels through activation-aware scaling [18], and HQQ optimizes quantization parameters without calibration data [19]. GPTAQ improves asymmetric calibration by incorporating information from the full-precision computation when calibrating the quantized model [20].

2.2. Learned Quantization Parameters

Optimizing a small set of quantization parameters can recover accuracy without updating all dense weights [22,32]. OmniQuant learns clipping thresholds and equivalent transformations with a differentiable, block-wise objective [23]. GSQ combines discrete assignment optimization with scale updates and also reports a fixed-assignment, end-to-end scale-only stage for 2-bit language models [24]. PiSO examines the structure of scale selection for round-to-nearest quantization and develops interval-based scale optimization [25].

GSQ combines discrete-assignment and scale learning with global scale-only distillation, while PiSO optimizes local activation-weighted reconstruction with scale-dependent rounding. OmniQuant learns clipping and equivalent transformations through block reconstruction. PSC applies global scale refinement to 2-bit vision encoders, using a frozen multimodal projector to supervise image-only calibration. Matched encoder-output and projected-feature controls isolate the effect of this supervision boundary.

2.3. Multimodal Quantization

Multimodal calibration introduces choices that do not arise in a text-only model, including which modalities supply calibration signals and which representation boundary is reconstructed. MBQ accounts for the differing sensitivities of visual and language tokens [33]. QIG uses quantization-aware integrated gradients to obtain finer-grained importance information for large vision-language models [34].

3. Methods

3.1. Problem Setup and Quantized Representation

Let $E_W$ denote the vision encoder with weights W, P the multimodal projector, and D the language decoder. An image x is processed into the crops required by the model’s standard any-resolution (AnyRes) pipeline. The encoder and projector produce visual tokens, which are subsequently merged and unpadded before being combined with text tokens for decoding [2]. PSC targets the weight matrices in the visual transformer blocks used by this path. The projector and language decoder remain frozen.

For layer ℓ, let $W_{\mathsf{\ell }}\in {\mathbb{R}}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$. We use asymmetric group-wise quantization with group size $G=128$. In the column order used by GPTQ, each quantized weight is reconstructed according to Equation (1):

$${\tilde{W}}_{\mathsf{\ell },o,k}=S_{\mathsf{\ell },o,g\left(k\right)}\left(Q_{\mathsf{\ell },o,k}-Z_{\mathsf{\ell },o,g\left(k\right)}\right),$$

(1)

where $Q_{\mathsf{\ell },o,k}\in \{0,\dots ,2^b-1\}$ is an integer code, Z is a zero point, $S>0$ is a scale, and $g\left(k\right)$ identifies the group of column k [35]. The inverse column permutation restores the matrix used by the model. In the main experiments, $b=2$: the targeted layers use 2-bit weights and BF16 activations.

3.2. Range-Searched GPTQ Initialization

We construct a strong GPTQ initialization using the candidate range multipliers $\mathcal{R}=\{1.0,0.9,0.8\}$. Each candidate runs the GPTQ error-compensation procedure. The min–max candidate corresponds to $r=1.0$.

Let ${\mathbf{w}}_{\mathsf{\ell },o}$ and ${\widehat{\mathbf{w}}}_{\mathsf{\ell },o}^{\left(r\right)}$ denote an original output row and its reconstructed candidate, written as column vectors. Let $H_{\mathsf{\ell }}$ be the input Gram matrix estimated from that layer’s calibration inputs, before damping. The implementation uses the damped matrix inside GPTQ’s error-compensation solve, but the original Gram matrix for candidate scoring. The selected range is given by Equation (2):

$$r_{\mathsf{\ell },o}^{★}=\arg \underset{r\in \mathcal{R}}{\min }{({\mathbf{w}}_{\mathsf{\ell },o}-{\widehat{\mathbf{w}}}_{\mathsf{\ell },o}^{\left(r\right)})}^{\mathsf{T}}{H}_{\mathsf{\ell }}({\mathbf{w}}_{\mathsf{\ell },o}-{\widehat{\mathbf{w}}}_{\mathsf{\ell },o}^{\left(r\right)}).$$

(2)

The comparison uses the BF16-dequantized candidate weights. For each output row, we retain the chosen candidate’s codes, scales, and zero points; the shared column permutation is unchanged. The damping setting is 0.01. We denote this three-range GPTQ configuration by GPTQ-R3 throughout the paper. Figure 1 summarizes the two-stage method.

Figure 1. Projected-feature scale calibration (PSC) has two stages. Range-searched GPTQ establishes the integer representation. Projected-feature calibration then jointly updates only its existing group scales. The mean squared error (MSE) loss is computed before any-resolution (AnyRes) merge and unpadding. The bfloat16 (BF16) teacher branch supplies fixed targets; gradients through the frozen student projector update the vision scales. The language decoder is used for downstream evaluation.

3.3. Joint Calibration of Existing Group Scales

Starting from the selected scales $S_{\mathsf{\ell },0}$, PSC introduces one log-scale update for each existing scale (Equation (3)):

$$S_{\mathsf{\ell }}\left({\mathrm{\Delta }}_{\mathsf{\ell }}\right)=S_{\mathsf{\ell },0}\odot \exp \left({\mathrm{\Delta }}_{\mathsf{\ell }}\right),{\mathrm{\Delta }}_{\mathsf{\ell }}{|}_{\mathrm{initial}}=0.$$

(3)

Substituting Equation (3) into Equation (1) yields the reconstructed weight in Equation (4):

$${\tilde{W}}_{\mathsf{\ell },o,k}\left({\mathrm{\Delta }}_{\mathsf{\ell }}\right)=S_{\mathsf{\ell },0,o,g\left(k\right)}\exp \left({\mathrm{\Delta }}_{\mathsf{\ell },o,g\left(k\right)}\right)\left(Q_{\mathsf{\ell },o,k}-Z_{\mathsf{\ell },o,g\left(k\right)}\right).$$

(4)

The exponential parameterization keeps every scale positive and makes the update relative to its own initialization. It also avoids assigning the same absolute step size to groups whose initial scales differ.

All ${\mathrm{\Delta }}_{\mathsf{\ell }}$ are optimized jointly through the complete visual path. Codes Q, zero points Z, and column permutations remain fixed. Biases, normalization parameters, embeddings, projector weights, and decoder weights are also frozen.

Group-level updates provide finer control than a single multiplier per module or output row. Each group shares a positive multiplier, preserving its code-centered weight direction while refining the scale. A strong integer initialization thus provides the foundation for calibration.

3.4. Projected-Feature Reconstruction Objective

For each calibration image, we cache the BF16 teacher features after the multimodal projector. Define the teacher and student projected features by Equation (5):

$$T_x=P\left(E_{W^{\mathrm{FP}}}\left(x\right)\right),F_x\left(\mathrm{\Delta }\right)=P(E_{\tilde{W}\left(\mathrm{\Delta }\right)}\left(x\right)).$$

(5)

Both tensors include the projected patch tokens from all crops of the image and are collected before AnyRes merge and unpadding. If $N_x$ is the number of such tokens and $d_P$ is the projected feature dimension, the objective in Equation (6) is

$${\mathcal{L}}_{\mathrm{PSC}}\left(\mathrm{\Delta }\right)={\displaystyle \frac{1}{\left|\mathcal{D}\right|}}\sum _{x\in \mathcal{D}}{\displaystyle \frac{\parallel F_x\left(\mathrm{\Delta }\right)-stopgrad\left(T_x\right){\parallel }_F^2}{N_x{d}_P}}.$$

(6)

Here, stopgrad marks the cached teacher target as constant. Normalizing within each image before averaging gives every calibration image equal weight across different crop counts.

The pre-merge objective yields a simple per-image target covering all crops. Merge and unpadding are differentiable, but change the positions and crop weights included in the loss. We compare both supervision locations experimentally.

The projector remains part of the differentiable student path even though its parameters are frozen. Consequently, a scale in an early visual block receives a gradient that depends on the later blocks and the projector. This is the central distinction from selecting a range using only a local layer reconstruction score. The objective directly matches visual representations in the language-aligned projection space.

4. Experimental Setup

4.1. Implementation Details

We evaluate LLaVA-OneVision with Qwen2 language backbones at the 0.5B and 7B scales [2,36]. Model-size labels refer to the language backbones. The vision encoder uses a model trained with sigmoid loss for language–image pre-training (SigLIP) [37]. We quantize the 156 linear weight matrices in the 26 visual transformer blocks covered by the selected visual path. The main experiments use asymmetric 2-bit vision weights with a group size of 128; activations, projector weights, and language-model weights remain in BF16. Main comparisons use a shared calibration sample; repeated-calibration controls use different random seeds. Section 5.3 extends the evaluation to CLIP/LLaVA-1.5 7B.

Each calibration or evaluation run uses one NVIDIA A100 graphics processing unit (GPU) with 80 GB of memory. The software environment used Python 3.10.20, PyTorch 2.6.0 with CUDA 12.4, Transformers 4.45.2, Accelerate 1.1.1, and NumPy 1.26.4. OneVision inference uses BF16, scaled dot-product attention, disabled TensorFloat-32 (TF32), the standard AnyRes image processor, greedy generation, and an image batch size of one. The CLIP extension retains BF16/greedy batch-one inference with its official padded single-crop processor. Further experimental settings are listed in Appendix A.

The calibration set contains 256 Common Objects in Context (COCO) images [38]. The default objective uses image pixels and teacher visual features only. After calibration, evaluation follows the full standard multimodal path, including merge, unpadding, text conditioning, and language decoding.

4.2. Optimization

PSC performs one epoch of 256 Adam [39] updates, using one image and all its crops per update. The learning rate is ${10}^{-3}$, the momentum coefficients are $(0.9,0.999)$, and the optimizer epsilon is ${10}^{-8}$. The global gradient norm is clipped to 1. Following every update, the relative scales are constrained as in Equation (7):

$$0.8\le S_{\mathsf{\ell },o,g}/S_{\mathsf{\ell },0,o,g}\le 1.25,\mathrm{or}\mathrm{equivalently}\log 0.8\le {\mathrm{\Delta }}_{\mathsf{\ell },o,g}\le \log 1.25.$$

(7)

The exported scales are $S_{\mathsf{\ell }}^{★}=S_{\mathsf{\ell },0}\odot \exp ({\mathrm{\Delta }}_{\mathsf{\ell }}^{★})$. They replace the existing scale tensors in the quantized checkpoint. Export preserves the original counts of codes, zero points, and scales. Algorithmic pseudocode and elementary properties of this parameterization are provided in Appendix G.

4.3. Evaluation

The main analysis focuses on TextVQA validation (5000 questions), OCRBench test (1000 questions), and ChartQA test (2500 questions) [11,12,13]. We report TextVQA accuracy, the OCRBench point score, and ChartQA relaxed overall accuracy. The supplementary evaluation includes GQA testdev-balanced, AI2D, ScienceQA-IMG, MME, and POPE [40,41,42,43,44]. Each benchmark retains its own metric.

TextVQA and GQA subsets informed method development, while a separate held-out TextVQA pool provided confirmation. Full-split scores include development examples; the data separation is described in Appendix A.6.

5. Results

5.1. Quantization Performance and Stability

PSC improves all three focal tasks over GPTQ-R3 at both OneVision model sizes and achieves the best scores among the tested 2-bit configurations (Table 1). Range search provides most of the initial TextVQA recovery over min–max GPTQ, with gains of 13.14 and 15.33 percentage points for 0.5B and 7B. PSC adds a further 2.83 and 2.85 percentage points without changing the chosen codes (Figure 2), showing that projected-feature calibration complements a strong discrete initialization.

Table 1. Text and chart understanding with 2-bit vision weights and BF16 language models. OCRBench is reported in points; TextVQA and ChartQA are percentages. Bold marks the best 2-bit value among the rows displayed for each model. Upward arrows (↑) indicate that higher scores are better; RTN denotes round-to-nearest quantization.

Method	TextVQA ↑	OCRBench ↑	ChartQA ↑
LLaVA-OneVision 0.5B
BF16 reference	65.93	585	61.48
RTN	6.83	12	10.44
HQQ	4.92	12	10.76
AWQ	19.81	69	25.04
GPTQ (min–max)	31.15	139	24.64
GPTAQ	42.40	238	45.00
GPTQ-R3	44.29	283	44.92
PSC	47.12	318	48.16
LLaVA-OneVision 7B
BF16 reference	76.03	620	79.92
RTN	6.45	19	14.84
HQQ	6.28	23	14.08
AWQ	24.98	91	46.32
GPTQ (min–max)	41.70	178	56.20
GPTAQ	54.60	311	69.44
GPTQ-R3	57.03	371	70.00
PSC	59.88	408	73.56

Figure 2. TextVQA accuracy for the BF16 reference, min–max GPTQ, range-searched GPTQ, and PSC. Quantized points use 2-bit vision weights and BF16 language models.

These improvements persist across different random seeds: PSC improves all three focal tasks, with a paired TextVQA gain of $2.81\pm 0.08$ percentage points (Table 2). All six main comparisons across the two model sizes and three tasks remain significant after Holm correction ($p\le 0.0015$; Table 3). The held-out TextVQA confirmation provides complementary evidence, yielding a 3.15-point gain with a 95% interval of [2.16, 4.16].

Table 2. Calibration variation on the 0.5B model with 2-bit vision weights: mean and sample standard deviation across different random seeds.

Task	GPTQ-R3	PSC	Paired Gain
TextVQA (%)	$45.00\pm 0.65$	$47.81\pm 0.61$	$2.81\pm 0.08$
OCRBench (points)	$286.00\pm 7.94$	$319.33\pm 3.21$	$33.33\pm 4.73$
ChartQA (%)	$46.45\pm 1.37$	$49.79\pm 1.64$	$3.33\pm 0.51$

Table 3. Paired PSC-minus-GPTQ-R3 gains with 2-bit weights.

Model	Task	Gain	95% Interval	Holm p
0.5B	TextVQA (pp)	+2.83	[+1.95, +3.72]	0.0006
0.5B	OCRBench (points)	+35.00	[+14.20, +56.17]	0.0015
0.5B	ChartQA (pp)	+3.24	[+1.87, +4.60]	0.0006
7B	TextVQA (pp)	+2.85	[+2.08, +3.62]	0.0006
7B	OCRBench (points)	+37.00	[+17.33, +57.20]	0.0012
7B	ChartQA (pp)	+3.56	[+2.40, +4.78]	0.0006

5.2. Ablation and Calibration Analysis

To identify the sources of this recovery, Table 4 compares initialization, update granularity, and supervision location at the same update budget. Starting from min–max GPTQ yields substantially lower scores, confirming the importance of a strong initialization. Group-scale updates also outperform row-shared updates on the text-centered tasks across different random seeds, by $0.59\pm 0.45$ percentage points on TextVQA and $4.33\pm 1.53$ points on OCRBench (Appendix F); GQA is less sensitive to granularity.

Table 4. Component controls for the 0.5B model with 2-bit vision weights. TextVQA and GQA are percentages; OCRBench is in points. Trainable values count relative scale updates. All rows use 256 images and 256 steps. Upward arrows (↑) indicate that higher scores are better.

Setting	Trainable Values	Steps	TextVQA ↑	OCRBench ↑	GQA ↑
PSC (default)	3,103,776	256	47.120	318	56.424
Min–max initialization	3,103,776	256	34.258	189	55.875
Module-shared update	156	256	43.342	281	56.201
Row-shared update	261,664	256	46.016	315	56.432
Local block reconstruction	3,103,776	256	43.798	274	56.297

Projected-feature supervision improves on local block reconstruction. A matched encoder-output control further isolates the supervision boundary: PSC gains 2.09 percentage points on TextVQA, 17 points on OCRBench, and 2.00 percentage points on ChartQA, with the TextVQA and ChartQA gains remaining significant after Holm correction. Post-unpadding supervision provides comparable point estimates, with no contrast significant after correction; we retain the pre-merge objective for its simple all-crop formulation. The matched comparisons appear in Appendix F, with additional loss and optimization controls in Appendix D.

Feature diagnostics show that this pre-merge calibration benefit persists at the assembled multimodal interface. On 512 diagnostic images, PSC reduces relative squared feature error after AnyRes merge and unpadding by 22.52% for 0.5B and 24.75% for 7B (Figure 3). The assembled features therefore remain closer to the BF16 reference, complementing the downstream task improvements.

Figure 3. Relative squared feature error at the actual AnyRes interface on a fixed 512-image diagnostic set. Lower is better.

Beyond the objective and parameterization, refinement-data composition also affects recovery (Appendix F). Replacing COCO refinement images with unlabeled TextVQA training images improves all three focal tasks, reaching 49.44% TextVQA, 365 OCRBench points, and 49.80% ChartQA; a mixed sample gives intermediate gains. At the same 256-update budget, 128 COCO images give similar scores to the 256-image default. These single-run controls show that domain-relevant images can further improve PSC while retaining image-only calibration.

5.3. Language Compression and Encoder Transfer

We next test whether the visual gains persist when the language model is also compressed. With the 7B language model quantized to 4-bit weights using RTN and a BF16 projector, PSC improves TextVQA, OCRBench, and ChartQA over GPTQ-R3 by 3.39 percentage points, 39 points, and 4.16 percentage points, respectively (Table 5). The gains in both language-precision settings demonstrate compatibility with language-model compression without including the decoder in calibration (Figure 4).

Table 5. Joint compression with the same 7B language checkpoint quantized to 4-bit weights by RTN. TextVQA and ChartQA are percentages; OCRBench is in points. The BF16-vision row isolates the effect of language-side compression. Upward arrows (↑) indicate that higher scores are better.

Language/Vision	TextVQA ↑	OCRBench ↑	ChartQA ↑
RTN/BF16 vision	75.19	617	79.28
RTN/GPTQ-R3 2-bit vision	55.48	366	68.32
RTN/PSC 2-bit vision	58.87	405	72.48

Figure 4. PSC gains over the matched GPTQ-R3 2-bit vision baseline for the 7B model, with either a BF16 or RTN 4-bit language checkpoint.

To examine transfer to another vision encoder, we apply PSC to CLIP ViT-L/14-336 [45] in LLaVA-1.5 7B [5], retaining the COCO256 sample and all optimization hyperparameters (Table 6). Without retuning, PSC improves CLIP TextVQA by 0.97 percentage points, with significance retained after Holm correction. OCRBench and ChartQA show smaller positive changes that are not significant. The TextVQA gain supports transfer of the calibration recipe to a second encoder.

Table 6. CLIP/LLaVA-1.5 7B results with 2-bit vision weights and a BF16 language model.

Method	TextVQA (%)	OCRBench (Points)	ChartQA (%)
BF16	45.89	318	18.08
GPTQ-R3	17.94	44	12.36
PSC	18.91	50	12.68

6. Storage, Calibration Cost, and Scope

PSC retains the compact quantized representation without adding scale entries or inference modules. The packed 2-bit vision encoder occupies 123.80 mebibytes (MiB) versus 758.70 MiB in BF16, a $6.13\times$ reduction (Table 7). With BF16 language weights, the corresponding full-model reductions are $1.59\times$ for 0.5B and $1.04\times$ for 7B.

Table 7. Packed-weight storage and TextVQA accuracy with BF16 language models and projectors.

Model	Vision	Vision MiB	Full MiB	Full Ratio	TextVQA (%)
0.5B	BF16	758.70	1704.53	1.00	65.93
0.5B	PSC 2-bit	123.80	1069.63	1.59	47.12
0.5B	PSC 3-bit	170.99	1116.82	1.53	63.45
0.5B	PSC 4-bit	218.18	1164.01	1.46	65.39
7B	BF16	758.70	15316.77	1.00	76.03
7B	PSC 2-bit	123.80	14681.87	1.04	59.88
7B	PSC 3-bit	170.99	14729.06	1.04	74.20
7B	PSC 4-bit	218.18	14776.25	1.04	76.04

Accuracy evaluation uses BF16-reconstructed weights. Inference speed with packed 2-bit weights remains to be evaluated.

Complete 0.5B calibration takes 728.28 s on one A100 versus 207.92 s for standard min–max GPTQ (3.50×), with 3.07 gibibytes (GiB) of peak allocated GPU memory. Measurement details appear in Appendix A.8.

Overall, 2-bit quantization minimizes vision-weight storage, while 3-bit and 4-bit weights recover more accuracy at 170.99 and 218.18 MiB per encoder. An accuracy gap to BF16 remains with 2-bit weights: 47.12% versus 65.93% on 0.5B TextVQA.

The experiments cover SigLIP and CLIP with multilayer perceptron (MLP) projectors. Extending calibration to interfaces such as TaiChi’s dual visual tokenizers and KAN-based projector for token communications [46] is a direction for future work.

7. Conclusions

This work addressed the preservation of fine-grained visual information when compressing the vision encoders of multimodal large language models to 2-bit weights. We proposed projected-feature scale calibration (PSC), which combines range-searched GPTQ initialization with joint refinement of existing group scales. Its central idea is to match teacher and student features through a frozen multimodal projector, aligning image-only calibration with the visual representation used by the language model. Integer codes and zero points remain fixed, and the learned updates fold into the original scales, preserving the compact representation and inference architecture. Experiments demonstrate consistent improvements in text recognition and chart understanding on LLaVA-OneVision, compatibility with language-model compression, and transfer to a CLIP-based encoder. PSC provides a practical approach to low-bit visual compression by preserving representations at the vision-to-language interface.

Appendix A.1. Models and Quantization

We use LLaVA-OneVision with Qwen2 0.5B and 7B language backbones [2] and a SigLIP vision encoder [37]. PSC targets 156 linear weight matrices across 26 vision transformer blocks, with 3,103,776 group scales. Quantization is asymmetric with group size 128. Unless 4-bit language-model compression is specified, activations and all non-target parameters, including the projector and language model, remain in BF16. Quantized vision weights are reconstructed in BF16 for evaluation.

Appendix A.2. Calibration

The default settings are listed in Table A1. Calibration uses 256 unlabeled COCO images [38], including all crops from the standard AnyRes processor. The target features are taken after the frozen projector, before merge and unpadding. The MSE is normalized by the number of projected tokens and feature dimensions for each image. PSC updates only the group scales; integer codes, zero points, and the group layout remain fixed. The final one-epoch model is used for every task. Repeated runs use different random seeds for COCO sampling and training order, with paired GPTQ initializations and fixed evaluation questions.

Table A1. Default calibration settings. BF16 denotes bfloat16 and FP32 denotes 32-bit floating-point precision.

Setting	Value
Vision weights/activations	2-bit/BF16
Quantization group size	128
Initialization	GPTQ-R3; ranges $\{1.0,0.9,0.8\}$
Range selection/damping	Per output row/0.01
Calibration data	256 unlabeled COCO images
Objective	Per-image normalized projected-feature MSE
Optimizer/learning rate	Adam/${10}^{-3}$
Batch size	One image with all AnyRes crops
Epochs/updates	1/256
Gradient clipping	Global norm 1
Relative scale bounds	$[0.8,1.25]$
Forward weights/scales	BF16/FP32

Table A1. Default calibration settings. BF16 denotes bfloat16 and FP32 denotes 32-bit floating-point precision.

Setting	Value
Vision weights/activations	2-bit/BF16
Quantization group size	128
Initialization	GPTQ-R3; ranges $\{1.0,0.9,0.8\}$
Range selection/damping	Per output row/0.01
Calibration data	256 unlabeled COCO images
Objective	Per-image normalized projected-feature MSE
Optimizer/learning rate	Adam/${10}^{-3}$
Batch size	One image with all AnyRes crops
Epochs/updates	1/256
Gradient clipping	Global norm 1
Relative scale bounds	$[0.8,1.25]$
Forward weights/scales	BF16/FP32

Appendix A.3. Implementation and Evaluation

Each run uses one NVIDIA A100 GPU (80 GB). We used PyTorch 2.6.0 with CUDA 12.4 and Transformers 4.45.2. Inference used BF16, greedy decoding, batch size one, and the Qwen1_5 conversation template.

Table A2 summarizes the benchmark splits and metrics. Each paired comparison uses the same evaluation inputs and decoding settings. No exact image overlap is found between calibration and evaluation sets. Development exposure and the held-out confirmation are described below.

Table A2. Evaluation benchmarks and metrics. Counts denote evaluated questions or items.

Benchmark	Split/Subset	Count	Metric
TextVQA [11]	Validation	5000	Accuracy (%)
OCRBench [12]	Test	1000	Points
ChartQA [13]	Test	2500	Relaxed overall accuracy (%)
GQA [40]	Testdev-balanced	12,578	Accuracy (%)
AI2D [41]	Test	3088	Accuracy (%)
ScienceQA-IMG [42]	Image-containing test subset	2017	Exact match (%)
MME [43]	Test	2374	Perception/cognition score
POPE [44]	Random, popular, adversarial	9000	F1 (%)

Table A2. Evaluation benchmarks and metrics. Counts denote evaluated questions or items.

Benchmark	Split/Subset	Count	Metric
TextVQA [11]	Validation	5000	Accuracy (%)
OCRBench [12]	Test	1000	Points
ChartQA [13]	Test	2500	Relaxed overall accuracy (%)
GQA [40]	Testdev-balanced	12,578	Accuracy (%)
AI2D [41]	Test	3088	Accuracy (%)
ScienceQA-IMG [42]	Image-containing test subset	2017	Exact match (%)
MME [43]	Test	2374	Perception/cognition score
POPE [44]	Random, popular, adversarial	9000	F1 (%)

Appendix A.4. CLIP Configuration and Matched Controls

LLaVA-1.5 7B uses CLIP ViT-L/14-336 penultimate-layer patch features, padded single-crop processing and a frozen two-layer MLP projector. We quantize the 138 linear matrices in its 23 active blocks; all other weights remain BF16. The COCO256 sample, optimizer, and update budget are unchanged from OneVision. Comparisons are within each checkpoint, since preprocessing and language backbones differ.

Supervision controls share PSC’s initialization, trainable group scales, image order, and 256 updates. The encoder-output control reconstructs features before the projector; the post-unpadding control reconstructs assembled visual tokens, including the fixed newline positions in the normalization. Data controls use 128 COCO images for two epochs, 256 unlabeled TextVQA training images, or a COCO128/TextVQA128 mixture. All retain the COCO256-derived GPTQ-R3 initialization and exclude task answers. Module- and row-shared controls tie relative scale multipliers without reducing the exported scale count. The historical 128-image/128-step row changes both data and compute; its missing ChartQA evaluation is denoted by a dash.

Appendix A.5. Baseline Adaptations

RTN and HQQ are data-free; other methods use the same COCO256 sample. HQQ uses its proximal solver with axis 1, group size 128, and non-rounded zero points. AWQ uses 20 activation-scale candidates, a 20-point clipping grid with maximum shrinkage 0.5, and 512 clipping tokens; query/key clipping is skipped. Supported reparameterizations are fused, while post-GELU inverse scaling is retained. GPTAQ uses correction coefficient 0.25, range multiplier 0.8 and the paired GPTQ column permutation. We retain these outer settings and each method’s native optimization procedure.

Appendix A.6. Evaluation and Data Separation

Evaluation uses greedy batch-one decoding with the official conversation template for each checkpoint. Maximum generation lengths are 1024, 128, and 16 tokens for TextVQA, OCRBench, and ChartQA, respectively. TextVQA supplies the image and question without OCR transcripts. TextVQA and ChartQA request short answers; OCRBench uses its dataset questions. Prompts and scorers are identical within each comparison.

Development used 512 TextVQA images (809 questions) and 512 GQA questions from 271 images. These observations informed method selection, but supplied no calibration targets. A separate TextVQA pool of 2522 images (3973 questions) was held out before PSC development, with the model pair and success criterion fixed before evaluation. GPTQ-R3 and PSC score 44.28% and 47.43%, meeting the criterion of a gain of at least one percentage point and a positive lower 95% confidence bound. This held-out comparison provides independent confirmation; the full-split results also contain development and confirmation examples. Exact calibration–evaluation image overlap is zero for all calibration samples and all eight benchmarks.

Appendix A.7. Statistical Analysis

Repeated-calibration tables report means and sample standard deviations over three runs with different random seeds and fixed evaluation questions. For fixed-checkpoint comparisons, we use image-cluster bootstrap intervals and two-sided paired randomization tests with 10,000 draws. Holm correction is applied separately to the six main OneVision contrasts, six supervision-control contrasts, and three CLIP contrasts. Pointwise intervals describe task uncertainty conditional on the evaluated checkpoints. OCRBench intervals are expressed in score points. The 7B and CLIP studies each use one calibration seed.

Appendix A.8. Storage and Cost Measurement

Storage includes packed integer codes, FP32 scales and zero points, int64 permutations, non-target BF16 weights, and serialization metadata. Each shared weight shard is counted once; unchanged tokenizer and configuration files are excluded. MiB denotes $2^{20}$ bytes. Calibration cost is measured in one sequential 0.5B run on an otherwise idle A100. The interval includes model loading, min–max GPTQ, R3 search, image-only teacher caching, and PSC refinement/export; model and dataset downloads are excluded. Peak memory denotes allocated GPU memory.

Appendix C.1. 3-Bit and 4-Bit Results

Table A5 and Table A6 compare GPTQ-R3 and PSC with 3-bit and 4-bit weights, with group size 128 and BF16 language models. These bitwidths have lower initial quantization error than 2-bit quantization, and the task-level changes are smaller and vary across benchmarks.

Table A5. Complete 3-bit vision results.

Model/Method	TVQA	OCR	CQA	GQA	AI2D	SQA	MME-P	MME-C	P-F1
0.5B GPTQ-R3	63.31	554	58.88	57.93	56.35	65.20	1215.09	243.57	86.06
0.5B PSC	63.45	551	59.56	57.70	56.22	65.34	1235.53	246.79	86.11
7B GPTQ-R3	73.44	587	79.16	61.81	80.57	94.74	1546.49	395.71	88.04
7B PSC	74.20	589	79.36	62.03	81.22	95.24	1558.93	380.36	88.63

Table A5. Complete 3-bit vision results.

Model/Method	TVQA	OCR	CQA	GQA	AI2D	SQA	MME-P	MME-C	P-F1
0.5B GPTQ-R3	63.31	554	58.88	57.93	56.35	65.20	1215.09	243.57	86.06
0.5B PSC	63.45	551	59.56	57.70	56.22	65.34	1235.53	246.79	86.11
7B GPTQ-R3	73.44	587	79.16	61.81	80.57	94.74	1546.49	395.71	88.04
7B PSC	74.20	589	79.36	62.03	81.22	95.24	1558.93	380.36	88.63

Table A6. Complete 4-bit vision results.

Model/Method	TVQA	OCR	CQA	GQA	AI2D	SQA	MME-P	MME-C	P-F1
0.5B GPTQ-R3	65.21	571	61.24	57.91	56.83	66.24	1233.58	255.00	86.42
0.5B PSC	65.39	579	61.44	57.87	56.70	66.04	1220.80	251.79	86.59
7B GPTQ-R3	75.54	615	80.28	62.07	80.93	95.74	1585.76	434.29	88.07
7B PSC	76.04	617	80.00	62.12	81.09	95.93	1580.16	437.86	88.17

Table A6. Complete 4-bit vision results.

Model/Method	TVQA	OCR	CQA	GQA	AI2D	SQA	MME-P	MME-C	P-F1
0.5B GPTQ-R3	65.21	571	61.24	57.91	56.83	66.24	1233.58	255.00	86.42
0.5B PSC	65.39	579	61.44	57.87	56.70	66.04	1220.80	251.79	86.59
7B GPTQ-R3	75.54	615	80.28	62.07	80.93	95.74	1585.76	434.29	88.07
7B PSC	76.04	617	80.00	62.12	81.09	95.93	1580.16	437.86	88.17

Appendix C.2. Interface Feature Error

Table A7 measures relative squared feature error on a fixed 512-image analysis set. Features are measured at the multimodal interface after AnyRes merge and unpadding, and results are averaged over images. A positive reduction indicates lower error under PSC.

Table A7. Relative squared feature error at the multimodal interface. Lower error is better. The reduction is $100(e_{\mathrm{R}3}-e_{\mathrm{PSC}})/e_{\mathrm{R}3}$.

Model	Vision	GPTQ-R3	PSC	Reduction (%)
0.5B	2-bit	0.426717	0.330609	22.52
0.5B	3-bit	0.169128	0.140691	16.81
0.5B	4-bit	0.068345	0.065319	4.43
7B	2-bit	0.362576	0.272839	24.75
7B	3-bit	0.124667	0.103895	16.66
7B	4-bit	0.045943	0.045943	0.00

Table A7. Relative squared feature error at the multimodal interface. Lower error is better. The reduction is $100(e_{\mathrm{R}3}-e_{\mathrm{PSC}})/e_{\mathrm{R}3}$.

Model	Vision	GPTQ-R3	PSC	Reduction (%)
0.5B	2-bit	0.426717	0.330609	22.52
0.5B	3-bit	0.169128	0.140691	16.81
0.5B	4-bit	0.068345	0.065319	4.43
7B	2-bit	0.362576	0.272839	24.75
7B	3-bit	0.124667	0.103895	16.66
7B	4-bit	0.045943	0.045943	0.00

The largest relative error reductions occur with 2-bit weights: 22.52% for 0.5B and 24.75% for 7B. Reductions are smaller with 3-bit and 4-bit weights, with negligible change for the 7B model with 4-bit vision weights. Feature-error reductions do not consistently translate into higher task scores at these bitwidths.

Appendix D.1. Scale Granularity, Loss, and Optimization

Table A8 compares calibration variants for the 0.5B model with 2-bit vision weights. Module-shared and row-shared updates use 156 and 261,664 trainable values, respectively, while retaining the original group scales at export. Local block reconstruction changes the supervision location under the same calibration budget. The 128-image variant uses 128 updates, reducing both the calibration data and the optimization budget.

Table A8. Calibration controls for the 0.5B model with 2-bit vision weights. TextVQA and GQA are percentages; OCRBench is in points.

Setting	Trainable Values	Steps	TextVQA	OCRBench	GQA
PSC (default)	3,103,776	256	47.120	318	56.424
Min–max initialization	3,103,776	256	34.258	189	55.875
Module-shared update	156	256	43.342	281	56.201
Row-shared update	261,664	256	46.016	315	56.432
Local block reconstruction	3,103,776	256	43.798	274	56.297
MAE objective	3,103,776	256	47.332	323	56.495
128 images/128 steps	3,103,776	128	46.426	320	56.400
Unbounded scales	3,103,776	256	47.120	318	56.424
Re-rounding control	3,103,776	256	47.120	318	56.424

Table A8. Calibration controls for the 0.5B model with 2-bit vision weights. TextVQA and GQA are percentages; OCRBench is in points.

Setting	Trainable Values	Steps	TextVQA	OCRBench	GQA
PSC (default)	3,103,776	256	47.120	318	56.424
Min–max initialization	3,103,776	256	34.258	189	55.875
Module-shared update	156	256	43.342	281	56.201
Row-shared update	261,664	256	46.016	315	56.432
Local block reconstruction	3,103,776	256	43.798	274	56.297
MAE objective	3,103,776	256	47.332	323	56.495
128 images/128 steps	3,103,776	128	46.426	320	56.400
Unbounded scales	3,103,776	256	47.120	318	56.424
Re-rounding control	3,103,776	256	47.120	318	56.424

The mean absolute error (MAE) objective yields slightly higher scores than mean squared error (MSE) in this comparison; MSE remains the default for the main experiments. The re-rounding control uses a piecewise gradient approximation. Both the unbounded-scale and re-rounding variants produce the same exported weights and scores as default PSC in this run, so these controls do not establish a benefit from scale bounds or fixed rounding.

Appendix D.2. A Language-Supervised Alternative

Table A9 compares projected-feature MSE with answer-distribution Kullback–Leibler (KL) supervision. The KL control requires language-side targets and decoder computation, whereas PSC uses only projected image features. KL supervision yields higher TextVQA and OCRBench scores but lower GQA accuracy than PSC in this setting.

Table A9. Image-feature and answer-distribution supervision for the 0.5B model with 2-bit vision weights.

Configuration	TextVQA (%)	OCRBench (Points)	GQA (%)
GPTQ-R3	44.29	283	56.05
PSC: projected MSE	47.12	318	56.42
Answer-logit KL control	48.24	342	55.98

Table A9. Image-feature and answer-distribution supervision for the 0.5B model with 2-bit vision weights.

Configuration	TextVQA (%)	OCRBench (Points)	GQA (%)
GPTQ-R3	44.29	283	56.05
PSC: projected MSE	47.12	318	56.42
Answer-logit KL control	48.24	342	55.98

Appendix G.1. PSC Procedure

Algorithm A1 summarizes PSC using the notation of the main paper. Initialization applies GPTQ error compensation [17] to each range candidate, followed by row-wise selection.

Algorithm A1. Projected-feature scale calibration.
Inputs: BF16 encoder E, frozen projector P, 256 calibration images, $\mathit{G}=128$, $\mathcal{R}=\{1.0,0.9,0.8\}$.
1	Run GPTQ for each range candidate and select the lowest second-order reconstruction score for each output row. Store Q, Z, $S_0$, and the column permutation.
2	Compute and cache $T_x=P\left(E_{W^{\mathrm{FP}}}\left(x\right)\right)$ for all calibration images using every AnyRes crop, before merge and unpadding.
3	Create ${\mathrm{\Delta }}_{\mathsf{\ell }}=0$ for every target group scale. Freeze all other variables. Initialize Adam with the settings in Table A1.
4	Shuffle the calibration images. For each image x, reconstruct the student weights from $S_0\odot e^{\mathrm{\Delta }}$, Q, and Z, then restore the original column order.
5	Evaluate $F_x\left(\mathrm{\Delta }\right)=P(E_{\tilde{W}\left(\mathrm{\Delta }\right)}\left(x\right))$ and compute $\parallel F_x\left(\mathrm{\Delta }\right)-T_x{\parallel }_F^2/\left(N_x{d}_P\right)$.
6	Backpropagate through the frozen projector and the complete visual encoder. Clip the global gradient norm to 1 and update all ${\mathrm{\Delta }}_{\mathsf{\ell }}$ with Adam.
7	Clamp ${\mathrm{\Delta }}_{\mathsf{\ell }}$ elementwise to $[\log 0.8,\log 1.25]$. Continue until the one-epoch endpoint.
8	Export $S_{\mathsf{\ell }}^{★}=S_{\mathsf{\ell },0}\odot e^{{\mathrm{\Delta }}_{\mathsf{\ell }}}$ into the existing scale entries and discard $\mathrm{\Delta }$. Retain Q, Z, the permutation, and all non-target model parameters.

Appendix G.2. Representation and Gradient Identities

The following identities apply to the reconstructed real-valued weights before finite-precision casting.

• Fixed-group direction. For a particular output row and group, let the centered code vector be $\mathbf{c}=\mathbf{q}-z\mathbf{1}$. Its reconstructed weight vector is

  $$\tilde{\mathbf{w}}\left(\delta \right)=s_0{e}^{\delta }\mathbf{c}=e^{\delta }\tilde{\mathbf{w}}\left(0\right).$$

  (A1)

Equation (A1) shows that the update preserves the group’s direction, signs, and code-centered zeros. A group with $\mathbf{c}=0$ cannot be changed by scaling. This restricted search space makes the quality of the discrete initialization important.

• Scale gradient. For a differentiable loss $\mathcal{L}$ on the reconstructed model,

  $${\displaystyle \frac{\partial \mathcal{L}}{\partial {\mathrm{\Delta }}_{\mathsf{\ell },o,g}}}=\sum _{k:g\left(k\right)=g}{\displaystyle \frac{\partial \mathcal{L}}{\partial {\tilde{W}}_{\mathsf{\ell },o,k}}}{\tilde{W}}_{\mathsf{\ell },o,k}.$$

  (A2)

Equation (A2) follows from $\partial {\tilde{W}}_{\mathsf{\ell },o,k}/\partial {\mathrm{\Delta }}_{\mathsf{\ell },o,g}={\tilde{W}}_{\mathsf{\ell },o,k}$ for the associated group. Gradients pass through the frozen projector even though its parameters are not updated.

• Export equivalence. Define the exported scale as $s^{★}=s_0{e}^{{\delta }^{★}}$. Substituting it into the original dequantizer yields

  $$s^{★}(q-z)=s_0{e}^{{\delta }^{★}}(q-z),$$

  (A3)

  Equation (A3) provides the same reconstructed weight without changing the number of scale entries, codes, or zero points.

Appendix G.3. Why the Projection Boundary Changes the Objective

For a small encoder-output perturbation $\epsilon$, a local linearization of the frozen projector yields

$$P(h+\epsilon )-P\left(h\right)\approx J_P\left(h\right)\epsilon ,{\parallel P(h+\epsilon )-P\left(h\right)\parallel }_2^2\approx {\epsilon }^{\mathsf{T}}{J}_P{\left(h\right)}^{\mathsf{T}}{J}_P\left(h\right)\epsilon .$$

(A4)

In Equation (A4), the matrix $J_P{\left(h\right)}^{\mathsf{T}}{J}_P\left(h\right)$ weights encoder-output perturbations according to their effect on the projected features. This local relation explains how the projector changes the calibration objective.