Exploring Selective Layer Freezing Strategies in Transformer Fine-Tuning: NLI Classifiers with Sub-3B Parameter Models

Abstract

In recent years, methods that selectively fine-tune or reduce the number of layers in large language models (LLMs) have garnered attention as an efficient alternative to traditional fine-tuning, where all layers are trained. In this study, we revisit the classical concept of layer freezing and propose a simple, effective strategy that selectively fine-tunes only a portion of transformer layers. We show that freezing the bottom 25% or 50% of layers in small-scale LLMs with sub-3 billion parameters yields significant improvements in memory efficiency and training speed while maintaining, or even surpassing, the performance of full fine-tuning and Low-Rank Adaptation (LoRA). Through experiments on Natural Language Inference (NLI) tasks using LLMs with fewer than 3 billion parameters, our approach achieves up to 50% memory savings and 30% faster training. Notably, our method does not require architectural modifications or additional parameters, making it particularly suitable for resource-constrained environments.

1. Introduction

Recent advancements in Natural Language Processing (NLP) have been largely driven by the emergence of Large Language Models (LLMs) such as GPT-3 [1], PaLM [2], and LLaMA [3]. These models leverage extensive pre-training to learn a wide range of linguistic patterns and knowledge. They demonstrate high performance across various tasks through techniques like prompt engineering [4] and in-context learning [5]. As a result, LLMs have become indispensable tools in numerous NLP tasks, including translation, question answering, and document generation.

However, several limitations are associated with the training and application of LLMs: First, high-performing LLMs typically contain over 7 billion parameters, making them computationally expensive and requiring vast resources and time for both training and inference. Additionally, LLMs tend to exhibit inconsistent performance in unfamiliar domains or tasks that were not encountered during pre-training [6]. This is a chronic problem for pre-trained models, which are constrained to generating outputs based on their pre-trained knowledge. To address this fundamental issue, it is imperative to implement knowledge updates through fine-tuning processes, enhancing the model’s adaptability [7]. However, due to the immense size of modern LLMs, with parameter counts ranging from billions to hundreds of billions, even fine-tuning demands significant computational effort.

To address these computational challenges, we revisit the concept of Layer Freezing, a simple yet effective fine-tuning strategy, and introduce detailed strategies that extend this approach. Previous studies have explored freezing layers of small language models such as BERT [8] during fine-tuning, but these efforts have mainly focused on improving speed and have encountered challenges due to the complexity of freezing techniques [9,10].

In contrast, we have found that simply freezing a subset of layers can achieve better computational efficiency and superior performance compared with fine-tuning the entire model. Rather than introducing additional layers or parameters, our goal is to reduce costs and maximize training efficiency by focusing on fine-tuning only a subset of layers within the existing LLM. The discovered method has the following advantages:

• Simplicity: This approach is highly straightforward and can be easily applied without the need for complex analysis or modifications to the model architecture.
• Universality: This method can be widely applied across various model architectures, regardless of scale or structural complexity.
• Performance Improvement: Experimental results show that this method not only improves computational efficiency but also enhances model performance compared with fine-tuning all layers.

Figure 1 illustrates a comparison between the conventional fine-tuning method that utilizes all layers and the layer selection method tested in this study. In the Fixed Freeze scenario, a predetermined ratio and location of layers are frozen before training begins. In contrast, the Adapted Freeze approach involves recording and analyzing the initial training process, selecting the appropriate layers to freeze, and then completing the remaining training.

To validate our approach, we focused on LLMs with fewer than 3 billion parameters, which can be trained on a single GPU. We conducted a comparative analysis using the Natural Language Inference (NLI) task [11], a subset of text classification problems. The NLI task was chosen as it effectively assesses a model’s fundamental language understanding and reasoning capabilities.

In our experiments, we found that freezing the bottom 25% of transformer layers during fine-tuning yields significant improvements in both computational efficiency and model performance. This simple approach reduced training memory usage by over 30% (excluding the static model parameters) compared with full-model fine-tuning, while generally improving overall performance. Additionally, we observed approximately a 20% increase in training speed. Empirical comparisons revealed that our layer-freezing strategy achieved superior performance metrics relative to Low-Rank Adaptation (LoRA) [12], while offering similar gains in training speed and memory reduction. These findings demonstrate that a conceptually simple, classical fine-tuning technique can be repurposed to meet the demands of modern LLMs while achieving competitive performance.

2. Related Work

2.1. Natural Language Inference

NLI, also known as recognizing textual entailment [13], is a fundamental task in NLP. This task involves determining the logical relationship between a premise and a hypothesis. Specifically, a model must classify this relationship as entailment (the hypothesis necessarily follows from the premise), contradiction (the hypothesis contradicts the premise), or neutral (the hypothesis may or may not be true given the premise).

NLI serves as a crucial indicator for evaluating a model’s language understanding and reasoning capabilities, including comprehension of semantics, context, and logical relationships. Models that demonstrate high performance in NLI tasks tend to excel in other language understanding tasks as well. Success in NLI challenges often indicates a level of language understanding that generalizes across various domains and linguistic tasks [14]. Consequently, NLI has established itself as a valuable benchmark for assessing language models [15].

2.2. General Fine-Tuning Approaches

The traditional fine-tuning approach involves retraining all or some of the parameters of a pre-trained model to adapt it to a new task [16]. This method adjusts the weights of the pre-trained model, utilizing various strategies such as learning rate adjustment, gradual unfreezing, and discriminative learning rates [17]. While this is useful for training task-specific models, it has limitations in terms of computational cost and resource efficiency when fine-tuning all parameters in recent large-scale models. To address these issues, techniques that reduce the number of trainable parameters, such as LoRA, or reduce memory usage, such as LLM quantization [18], are gaining attention.

2.3. Parameter-Efficient Fine-Tuning Approaches

2.3.1. Layer Freezing

Since the emergence of language models, there have been attempts to improve the efficiency of fine-tuning by freezing certain layers. Ref. [9] proposed a method that fine-tunes only the bias parameters instead of the weights in transformer-based masked language models, reducing memory usage and improving speed. Ref. [10] introduced a technique to accelerate the training process by gradually freezing layers based on their impact during training. However, both studies primarily focus on speed improvement, which often leads to performance degradation. Additionally, these studies require complex mechanisms for layer freezing.

Recent work revisits layer freezing for LLM fine-tuning with modern strategies, offering different perspectives on parameter selection and freezing mechanisms. Ref. [19] proposes Half Fine-Tuning (HFT), which randomly freezes 50% of parameters each round across all transformer layers, retaining pre-trained knowledge and mitigating catastrophic forgetting. HFT achieves comparable or better task performance with around a 30% reduction in training time by treating parameter selection as a regularization mechanism. However, HFT’s random selection approach may freeze task-critical parameters, potentially limiting its effectiveness for specific tasks requiring targeted layer adaptation. Ref. [20] introduces SAFE (Selective Adapter Freezing Early), which selectively freezes less important adapter modules early in training based on centered kernel alignment importance scores. SAFE reduces memory use by 43%, computation by 35%, and training time by 12% without degrading accuracy. While SAFE demonstrates sophisticated importance-based selection, it requires additional computational overhead for importance scoring and is limited to adapter-based fine-tuning architectures.

These selective freezing techniques demonstrate that layer- or module-level freezing can yield substantial efficiency gains. However, they differ fundamentally in their selection mechanisms: HFT employs stochastic selection for simplicity, SAFE uses importance-based selection for precision, while our approach utilizes positional selection based on transformer layer hierarchy. Unlike HFT’s random approach that may inadvertently freeze crucial upper layers needed for task-specific reasoning in NLI tasks, our bottom-up strategy systematically preserves lower-layer linguistic features while allowing upper layers to adapt to logical inference requirements. Compared with SAFE’s adapter-module focus requiring architectural modifications, our method operates directly on transformer layers without additional components, making it more suitable for resource-constrained environments with small language models where adapter overhead may be prohibitive.

2.3.2. Low-Rank Adaptation

LoRA was first introduced by Ref. [12] as a parameter-efficient fine-tuning method for large pre-trained language models. Rather than updating the full set of model parameters, LoRA injects trainable low-rank matrices into the attention layers of transformer architectures while keeping the original weights frozen. This reduces the number of trainable parameters and the overall memory footprint, enabling efficient adaptation with minimal resource requirements. LoRA maintains model expressiveness by learning task-specific directions in a low-dimensional subspace.

Recently, Ref. [21] proposes AdaLoRA, which adaptively allocates low-rank update parameters based on importance, resulting in better performance under tight parameter budgets. Ref. [22] introduces QLoRA, which combines 4-bit quantization of the base model with LoRA adapters, enabling efficient fine-tuning of very large models on a single 48 GB GPU without performance loss. More recently, Ref. [23] proposes GeoLoRA, a dynamic low-rank training method based on matrix differential equations that adaptively allocates parameter budget across layers via geometric principles. GeoLoRA achieves improved convergence guarantees and outperforms AdaLoRA in both accuracy and computational efficiency. These advancements demonstrate the continued evolution of LoRA-based adaptation techniques, improving the efficiency-accuracy trade-off for modern large language models.

In contrast, our study employs a simple freezing method that can be applied to any model while also demonstrating performance improvement. This straightforward approach stands out in that it not only reduces computational cost but also enhances model performance.

3. Layer Selection for Fine-Tuning

This study proposes a method to improve model training efficiency by fine-tuning only a subset of layers in an LLM and aims to validate it experimentally. In contrast to conventional LLM fine-tuning methods that involve training all layers, this study demonstrates that selectively freezing specific layers and fine-tuning only the remaining ones can lead to improved performance, faster learning speed, and reduced memory usage.

3.1. Fixed Freezing

The main strategy used during the model training process is to freeze specific layers of the model and fine-tune only the remaining layers. We evaluated the impact of various layer selection methods on model performance and training efficiency:

• Bottom-Up Freezing: We experimented with a method that sequentially freezes the model’s bottom layers, allowing only the remaining upper layers to be fine-tuned. As the bottom layers are primarily responsible for the basic linguistic expressiveness of the language model, while the upper layers tend to learn task-specific representations [24], we hypothesized that this freezing approach would preserve fixed linguistic knowledge while enabling task-specific adaptation.
• Top-Down Freezing: We tested an approach that freezes the top layers and trains only the bottom layers. This method anticipates that the bottom layers will be tuned to the task based on the fixed higher-level concepts in the frozen upper layers.
• Interval Freezing: This method involves freezing layers at intervals of n, meaning that every n-th layer is frozen during training. This approach aims to allow both upper and lower layers to be appropriately adjusted simultaneously, encouraging information to be learned evenly across various layer levels.

3.2. Adapted Freezing

As an alternative to the Fixed Freezing strategy, we propose an Adaptive Freezing approach with dynamic layer selection. In this approach, we track the weight changes of each layer during training to identify layers with significant or minimal changes. Based on these changes, we automatically identify the layers that play a crucial role in performance. The Top-N layers, according to the magnitude of weight changes, are then selectively frozen before proceeding with training. The following outlines the operational sequence of this adaptive layer selection method:

• Weight Change Tracking: We calculate the change in weights for each layer by comparing the layer-wise weights before training and after the first five steps of training. The magnitude of weight changes for multiple parameters within a single layer was quantified as a single scalar value by computing the L2 norm of the changes and then taking the mean across all parameters in the layer.
• Top-N Layer Selection: We select the top N layers with either the largest or smallest weight changes, freeze them, and then resume training. Through this, we aimed to understand the roles that layers with large and small weight changes play in the fine-tuning.

3.3. Freezing Strategies

Through these Fixed and Adapted Freezing methods, we aim to experimentally demonstrate that fine-tuning only a subset of layers can reduce memory and computational costs compared with fine-tuning the entire model. We hypothesize that this approach can potentially improve performance or, at a minimum, maintain it without degradation.

The following are abbreviations for the freezing strategies used in this study:

• ALL: Fine-tuning using all layers, used as the baseline.
• LoRA: Fine-tuning using LoRA, also used as the baseline.
• INT (Interval): Fine-tuning by freezing layers at regular intervals.
• BOT (Bottom Up): Fine-tuning by freezing layers starting from the bottom of the model.
• TOP (Top Down): Fine-tuning by freezing layers starting from the top of the model.
• ADT-L (Adapted Low): Fine-tuning by freezing N layers with the smallest weight changes.
• ADT-H (Adapted High): Fine-tuning by freezing N layers with the largest weight changes.

The number following each abbreviation indicates the percentage of frozen layers. For example, INT25 means 25% of the layers are frozen at regular intervals, while TOP50 means 50% of the layers are frozen starting from the top.

Figure 2 visualizes the freezing strategies utilized in this study. When 50% of the total layers are frozen, the layers are frozen and trained in the pattern shown in the figure for each strategy.

4. Experiments

The primary objective of this study is to verify whether fine-tuning only a subset of layers in LLMs can achieve sufficient training effectiveness compared with training all layers. Through our approach, we aim to explore a methodology that reduces memory and computational resource requirements while maintaining training effectiveness without performance degradation.

4.1. Models and Datasets

The experiments used decoder-only small LLMs with sub-3 billion parameters, such as Gemma-2b (Gemma) [25], Phi-2 [26], and MiniCPM-2b-128k (MiniCPM) [27], which can be trained on a single GPU. All these models are large-scale pre-trained language models whose parameters can be fine-tuned for specific tasks.

For the experiments, we used NLI tasks, which are primarily text classification problems designed to verify the models’ basic language understanding and reasoning abilities by determining logical relationships between sentences. NLI-related tasks from the GLUE [15] and SuperGLUE [28] benchmarks were used. The specific tasks are listed in

Table 1. Number of samples in each dataset; the number in parentheses is the total number of data, but we used only a maximum of 3200 randomly selected data.

Dataset	Task	Number of Samples	Usage Rate
GLUE	RTE	2500	100%
GLUE	QNLI	3200 (105,000)	3.0%
GLUE	WNLI	635	100%
GLUE	MNLI	3200 (392,000)	0.8%
SuperGLUE	CB	250	100%

.

4.2. Experimental Design

In this study, we conducted a series of experiments to evaluate the efficacy of fine-tuning strategies that selectively utilize specific layers of neural networks. Our experimental design focused on various freezing techniques, enabling a comparative analysis of performance variations resulting from each approach. The primary objective of these experiments was to conduct a comprehensive assessment of model performance, memory utilization, and training efficiency to determine the optimal freezing methodology.

Finding the Optimal Freezing Ratio: First, we conducted experiments to determine at which ratio of frozen layers the model exhibits the highest performance. To achieve this, we froze a certain proportion of the model’s layers and trained only the remaining layers. For the INT strategy, we applied freezing ratios of 25%, 33.3%, and 50%, while for the BOT, TOP, ADT-L, and ADT-H strategies, we used ratios of 25%, 50%, and 75%. Through this approach, we aimed to identify the threshold at which performance sharply declines when more than a certain proportion of layers are frozen, thereby deriving the optimal freezing ratio.

Finding the Optimal Freezing Position: Next, we analyzed which positions within the model’s layers have the most significant impact on performance when frozen. The freezing positions were categorized as BOT (Bottom Up; freezing layers sequentially starting from the lower layers), TOP (Top Down; freezing layers sequentially starting from the upper layers), and INT (Interval; freezing layers at regular intervals). By comparing the performance for each freezing position, we examined which layers play more crucial roles. Additionally, we tracked the weight changes during the training process and adaptively froze layers to understand the significance of the weight changes.

Finding the Optimal Freezing Strategy: Finally, we aimed to identify the optimal freezing strategy by comprehensively considering the performance, training speed, and memory usage of each strategy. As it is challenging to fairly and objectively quantify these diverse aspects into a single metric, we first evaluated each aspect quantitatively and then conducted a comprehensive analysis.

4.3. Hyper-Parameter Setting

For each strategy, we conducted training using five different random seeds: 42, 43, 44, 45, and 46, and measured the average performance. We fixed the batch size at 32 and max length at 128 and trained the model for a total of 100 steps for each experiment. The learning rate is $5\times e^{-5}$, and we used a cosine learning rate scheduler. For the LoRA implementation in our experiments, we set the rank (r) to 8 and the alpha parameter to 32. A dropout rate of $0.1$ was applied, and no quantization was performed. All other hyper-parameters remained consistent across experiments.

4.4. Performance Comparison by Freezing Strategy

Figure 3 shows the average performance of the three models on the QNLI task, measured five times using different random seeds for various freezing strategies. The error bars represent the $\mu \pm \sigma$ (mean ± standard deviation). The gray bars represent the experimental results for the baseline approaches: full fine-tuning ALL and LoRA. The red dashed line indicates the mean performance when all layers are fine-tuned. The blue dashed line represents the mean performance minus one standard deviation, while the green dashed line shows the mean performance plus one standard deviation when all layers are fine-tuned.

In most cases, the INT25, BOT25, and TOP25 strategies showed superior performance compared with the ALL and LoRA strategy. Notably, the Phi-2 model achieved even higher performance, especially with the freezing strategies. The BOT strategy consistently demonstrated excellent and solid performance across all models and most tasks. Furthermore, the INT strategy generally showed lower standard deviation compared with ALL, indicating more stable learning. Contrary to expectations, the Adapted Freezing strategy did not show performance merits. Furthermore, freezing layers with either high or low weight changes did not yield significant results.

Total experimental results can be found in Appendix A.

4.5. Training Efficiency Comparison by Freezing Strategy

Figure 4 shows the average performance and training time measured by training the Gemma model on the MNLI task for 10 steps using five different random seeds. The size of the circles visually represents the proportion of layers that were frozen. From this, it can be observed that the INT, BOT, and TOP strategies learn faster than the ALL strategy. In particular, the BOT strategy generally showed very fast learning speed, comparable to the TOP and INT strategies. Moreover, when compared with LoRA, the BOT strategies showed comparable or superior learning speed. ADT-L and ADT-H, on the other hand, exhibited relatively slower speeds due to an additional fixed time of about 13 s required for selecting the initial layers to freeze. With sufficiently long training times, this overhead becomes negligible, and their speed is expected to be comparable to that of the INT and TOP strategies. While Figure 4 shows Gemma’s efficiency results to avoid redundant visualizations, we conducted identical efficiency measurements for Phi-2 and MiniCPM and observed consistent trends. Additional experimental results can be found in Appendix A.

Figure 5 shows the GPU memory usage of the INT and BOT strategies compared with ALL. Excluding the memory used by the fixed model, when comparing only the training memory, the INT strategy used approximately 7–13% less memory and the BOT strategy used approximately 12–25% less. In the case of BOT, as the freeze ratio increased, the GPU memory usage dramatically decreased by about 20% at each ratio. Additionally, BOT25 demonstrated a level of memory reduction comparable to that of LoRA. The smaller reduction in INT is presumed to be due to inefficient computation caused by a lack of optimization during CUDA operations, depending on the location of the frozen layers.

4.6. Best Freezing Strategy

Ranking Score = Mean (reversed rank): To determine the optimal strategy, we assigned scores based on the performance rankings from our experiments. In this paper, a total of 17 experiments were conducted using ALL and LoRA strategy and 15 freezing strategies. Accordingly, for each task, the strategy that achieved the highest average performance was given 16 points, while the strategy with the lowest average performance received 1 point. This scoring was performed for each of the 15 experiments (3 models × 5 tasks), and the final average score was calculated.

Figure 6 shows the average scores measured in this manner, and it can be observed that BOT25 and TOP25 achieved the highest scores. Notably, the INT strategy outperformed the ALL strategy across all ratios. This confirms that strategies INT25, INT33, INT50, BOT25, BOT50, and TOP25 can be used as alternatives to the ALL strategy. Considering memory usage and training speed, the BOT25 and BOT50 strategies are judged to be the most effective.

5. Discussion

Our experimental findings indicate that partial layer freezing, a concept traditionally used for transfer learning, remains highly relevant and effective in the context of modern LLMs. By revisiting this idea through a contemporary lens, we demonstrate that selectively freezing transformer layers can achieve a favorable trade-off between computational efficiency and task performance. This observation aligns with theoretical insights from representational learning, where lower layers typically capture general syntactic or lexical features, while higher layers encode task-specific semantics.

Enhanced Performance: Specifically, we found that freezing the bottom 25% or 50% of transformer layers during fine-tuning not only maintained high performance but also often exceeded the results of full model fine-tuning and LoRA. This approach led to a substantial reduction in memory usage, approximately 30% and 50%, respectively, without compromising model effectiveness. Notably, the training speed increased by 20–30%, which can be attributed to the reduced computational load. We posit that this phenomenon may be attributed to the model’s capacity being disproportionately large relative to the complexity of the NLI task. This aligns with observations in techniques like LoRA, where freezing the majority of the model and training only a small number of additional parameters can lead to performance improvements.

Memory Reduction: The reduction in memory usage observed with our partial fine-tuning approach is logically consistent with the decreased number of trainable parameters. However, we noticed that interval freezing strategies, where layers are frozen in a distributed pattern throughout the model, did not yield significant memory savings compared with contiguous bottom-layer freezing. This suggests that contiguous freezing of layers is more beneficial for memory optimization, though we acknowledge that the underlying mechanisms require more rigorous investigation beyond our current empirical observations.

Learning Speed Improvements: While the speed improvements did not completely match the reduction in memory usage, the observed 20–30% increase in training speed is nonetheless significant. We attribute this to the substantial computational overhead inherent in processing large language models.

Moreover, our method does not require architectural modifications or additional trainable parameters, which stands in contrast to popular PEFT methods such as LoRA. Although LoRA-based methods are effective, they introduce additional components that require careful configuration and tuning. Our findings highlight the possibility of combining the strengths of both approaches, for instance, by applying LoRA to the unfrozen layers while freezing others.

This simplicity and adaptability make our method particularly suitable for constrained environments, such as on-device fine-tuning or edge applications. Future work could explore adaptive freezing strategies guided by gradient norms or weight change statistics, potentially in conjunction with other PEFT methods.

6. Conclusions

This study revisited the concept of layer freezing and demonstrated its effectiveness for sub-3 billion parameter language models on natural language inference (NLI) tasks. By freezing specific subsets of transformer layers, particularly the bottom 25% or 50% of layers, we achieved up to 50% memory savings and 20–30% faster training while maintaining or improving performance compared with full model fine-tuning and LoRA in our experimental setting.

Our approach offers practical advantages through its simplicity: it requires no architectural modifications, additional parameters, or complex selection mechanisms. The method can be readily applied to any transformer architecture, though effectiveness is demonstrated specifically within our experimental scope of sub-3B models and NLI tasks.

While we explored adaptive freezing strategies based on weight dynamics, these approaches proved less effective than simple positional heuristics in our experiments, reinforcing that straightforward strategies can be surprisingly effective for small language models in resource-constrained environments.

Our findings provide empirical evidence for the effectiveness of positional freezing strategies within clearly defined boundaries. Future work should validate applicability across larger models and diverse NLP tasks, as we make no claims about universal effectiveness beyond our experimental scope. We recommend that practitioners validate this approach for their specific use cases rather than assuming broad generalizability.

7. Limitation

This study demonstrated the efficiency of fine-tuning only certain layers in LLMs. However, the following limitations exist:

Experiments Limited to Small LLMs: This study primarily conducted experiments on small-scale LLMs with 3 billion parameters or fewer, such as Gemma, Phi-2, and MiniCPM. This choice was due to the experimental conditions set to enable training in a single GPU environment. Therefore, further research is needed to determine whether the proposed methodology demonstrates similar performance improvements and efficiency in extremely large models (such as PaLM and LLaMA). In extremely large models, memory requirements or training patterns may differ, necessitating experiments on these models to expand the scope of this research.

Dataset Limited to NLI Tasks: This study focused on Natural Language Inference (NLI) tasks, conducting experiments only on NLI-related datasets (such as RTE and CB) from GLUE and SuperGLUE benchmarks. While NLI tasks are specialized in evaluating a model’s ability to infer logical relationships, other types of tasks (e.g., text generation, question answering, translation, etc.) may have different model characteristics and learning requirements. Therefore, further experiments on diverse tasks and datasets are necessary to assess the effectiveness of the proposed fine-tuning method across a broader range of natural language processing tasks.

Heuristic-based Strategies: Our layer selection strategies were static and preconfigured. Although we also explored adaptation based on early training behavior, this method relies on heuristics and may not generalize well across tasks or architectures. More principled approaches, such as learning-to-freeze mechanisms or data-driven selection policies, are worth investigating in future work.

Limited Mechanistic Understanding: Our analysis remains primarily empirical. While we observe the effectiveness of bottom-layer freezing, we acknowledge that a deeper mechanistic understanding of why these strategies work requires more sophisticated analysis techniques such as representational probing and gradient flow analysis.

Potential Reduced Model Plasticity: Layer freezing inherently reduces model adaptability, which may be unsuitable for tasks requiring full representational capacity. Our approach may not be optimal for scenarios where maximum model flexibility is crucial.

To address these limitations, future work should validate the generalizability and efficiency of the proposed methodology across a range of larger LLMs and diverse tasks.