Enhancing Whisper Fine-Tuning with Discrete Wavelet Transform-Based LoRA Initialization

Abstract

In low-resource automatic speech recognition (ASR) scenarios, parameter-efficient fine-tuning (PEFT) has become a crucial approach for adapting large pre-trained speech models. Although low-rank adaptation (LoRA) offers clear advantages in efficiency, stability, and deployment friendliness, its performance remains constrained because random initialization fails to capture the time–frequency structural characteristics of speech signals. To address this limitation, this work proposes a structured initialization mechanism that integrates LoRA with the discrete wavelet transform (DWT). By combining wavelet-based initialization, a multi-scale fusion mechanism, and a residual strategy, the proposed method constructs a low-rank adaptation subspace that better aligns with the local time–frequency properties of speech signals. Discrete Wavelet Transform-Based LoRA Initialization (DWTLoRA) enables LoRA modules to incorporate prior modeling of speech dynamics at the start of fine-tuning, substantially reducing the search space of ineffective directions during early training and improving convergence speed, training stability, and recognition accuracy under low-resource conditions. Experimental results on Sichuan dialect speech recognition based on the Whisper architecture demonstrate that the proposed DWTLoRA initialization outperforms standard LoRA and several PEFT baseline methods in terms of character error rate (CER) and training efficiency, confirming the critical role of signal-structure-aware initialization in low-resource ASR.

1. Introduction

In recent years, large-scale pre-trained models have achieved remarkable advances in natural language processing [1,2], computer vision [3,4,5], and speech recognition [6,7]. These models exhibit strong transferability and generalization capabilities; however, adapting them to downstream tasks such as document parsing [8] through full fine-tuning (FT) imposes substantial demands on computational resources, memory, and energy consumption, making this approach particularly impractical in low-resource or edge scenarios. Low-resource ASR is particularly challenging due to the intrinsic properties of speech signals, including pronounced non-stationarity, multi-scale temporal structures, cross-speaker and dialectal variability, and transient acoustic events. These factors make it difficult for models to generalize from limited labeled data, further highlighting the need for efficient adaptation techniques.

Parameter-efficient fine-tuning (PEFT) alleviates this challenge by freezing most pre-trained parameters and updating only a small subset, typically less than 1% of the model, thereby significantly reducing training costs while preserving performance. Representative PEFT methods include adapters [9], prefix-tuning [10], and low-rank adaptation (LoRA) [11]. Among them, LoRA has been widely adopted because it requires no architectural modifications, introduces no additional inference latency, and remains simple to implement. Nevertheless, LoRA continues to encounter performance limitations in low-resource ASR tasks, primarily because random initialization fails to exploit the multi-scale time–frequency structure inherent in speech signals. Speech signals exhibit pronounced local stationarity, cross-scale correlations, and transient dynamics. Introducing such structural priors at the initialization stage can substantially improve the convergence efficiency and final performance of LoRA [12,13,14,15,16].

To address this issue, this paper proposes DWTLoRA (Discrete Wavelet Transform-Based LoRA Initialization), a structured initialization mechanism. DWTLoRA extracts local time–frequency features from speech signals through wavelet decomposition and constructs a low-rank adaptation subspace with multi-scale representational capacity, enabling LoRA to respond to the structural characteristics of speech at initialization and providing clearer optimization directions during the early stages of training. Unlike existing methods, this study explicitly designs the interpolation and projection of wavelet basis vectors to match the dimensionality of the Whisper model’s linear projection layers, and uses the normalized vectors to initialize LoRA parameters. Furthermore, the effects of different wavelet bases are systematically evaluated, demonstrating the effectiveness of DWTLoRA in low-resource ASR fine-tuning. The proposed approach introduces structural priors while preserving the original Whisper architecture, incurs no additional inference overhead, and provides an efficient and robust initialization strategy for low-resource ASR adaptation.

To validate the effectiveness of DWTLoRA, this study integrates the proposed approach into the Whisper model and conducts systematic experiments on a 200-h Sichuan dialect speech corpus. The results demonstrate that, compared with standard LoRA and other PEFT baselines, DWTLoRA achieves notable improvements in both character error rate (CER) and convergence speed.

The remainder of this paper is organized as follows. Section 2 reviews related work and introduces the DWTLoRA architecture. Section 3 presents the wavelet-based structured initialization algorithm. Section 4 reports experimental results and provides detailed analysis.

2. DWTLoRA Architecture and Related Work

This study integrates DWTLoRA into an enhanced version of the Whisper model to construct the Whisper-DWTLoRA framework. The Whisper architecture, illustrated in Figure 1, serves as a general-purpose model for speech recognition and understanding that robustly handles diverse languages, accents, acoustic conditions, and tasks. In this framework, input audio is first processed through a convolutional front-end, consisting of 1D convolution followed by a GELU activation, which extracts low-level acoustic features. These features are then passed to the encoder to capture contextual representations across the entire sequence. The decoder generates output tokens while attending to the encoder representations via cross-attention, enabling alignment between the encoded audio features and the generated text.

Figure 1. Neural network architecture of the Whisper model, consisting of a Transformer-based encoder and decoder.

2.1. DWTLoRA Architecture

To address the low convergence efficiency and insufficient training stability caused by random initialization in conventional LoRA-based speech fine-tuning, this work proposes a structured initialization mechanism based on the discrete wavelet transform, referred to as DWTLoRA. The overall workflow, shown in Figure 2, proceeds as follows. First, the system extracts features from the input speech signal and applies a discrete wavelet transform to obtain wavelet coefficients that characterize the multi-scale time–frequency structure of the signal. Next, these physically interpretable basis vectors construct and normalize a structured low-rank subspace, which initializes the parameters of the LoRA adapters. The method then generates the final adapter initial weights through projection and residualization. Finally, the system injects the initialized LoRA modules into the linear layers of the Whisper model and performs parameter-efficient fine-tuning without modifying the original inference architecture.

Figure 2. Overview of the proposed DWTLoRA framework integrated into Whisper. DWTLoRA operates during the initialization and training stage by constructing structured LoRA parameters using wavelet-based multi-scale representations.

Speech signals exhibit strong non-stationarity and hierarchical time–frequency organization, whereas conventional random initialization fails to provide LoRA with priors aligned to these properties. This mismatch is particularly detrimental under low-resource conditions, as it disperses optimization directions and slows convergence. To address this, the discrete wavelet transform (DWT) is employed to construct a structured initialization subspace that directly encodes prior knowledge of speech into the initial directions of LoRA parameter updates. Specifically, the input audio waveform is first converted into low-level features such as Mel or log-Mel spectrograms, which are then processed through a DWT using a set of wavelet bases $\left({\psi }_1,\dots ,{\psi }_r\right)$ to obtain the corresponding wavelet basis representation. In the DWTLoRA initialization module, the resulting basis vectors $B_0$ are normalized, followed by a projection and a residual update step to generate the low-rank matrices $A$ and $B$. These matrices define the initial weight update $\mathsf{\Delta }{W}_0$, which is subsequently applied to the linear layers of the Whisper encoder, providing a structured, multi-scale initialization that aligns with the temporal and spectral characteristics of speech.

In this study, DWT was implemented using the Daubechies wavelet family, with Daubechies-4 (db4) wavelets selected by default as the basis functions. The db4 wavelet achieves a favorable balance between temporal localization and frequency resolution, and its compact support and strong capability in modeling transient acoustic events have led to widespread use in speech and audio signal analysis. The use of an orthogonal wavelet basis ensures energy preservation during decomposition and facilitates the construction of a stable low-rank initialization subspace. Considering that the hidden layer dimensions of the Whisper model are typically much higher than the length of the original speech signals, the selected wavelet basis vectors are interpolated and projected to match the dimensionality of the model’s linear projection layers, and subsequently normalized for initializing the LoRA parameters. This procedure preserves the multiscale structural information inherent to the wavelet family while maintaining compatibility with the model architecture.

To assess the sensitivity of DWTLoRA to the choice of wavelet family, comparative experiments were conducted using different orthogonal wavelet bases, including Haar, Daubechies-2 (db2), Daubechies-4 (db4), and Symlet-4 (sym4). All experiments were performed under identical training configurations, with the only variation being the type of wavelet basis function used during initialization. The results indicate that DWTLoRA is not highly sensitive to the specific choice of wavelet family, and the model performance under different wavelet bases consistently outperforms the randomly initialized LoRA baseline. Among the tested wavelets, db4 and sym4 achieved slightly better performance in terms of CER, likely due to their stronger capability in capturing smooth temporal variations and local spectral transitions in speech signals. In contrast, although the Haar wavelet offers higher computational efficiency, its lower frequency resolution provides relatively limited structural prior information.

Overall, by introducing DWT-based wavelet bases to guide LoRA initialization, the proposed approach substantially reduces ineffective search directions during early training, leading to faster convergence, improved training stability, and stronger generalization performance during fine-tuning.

2.2. Related Work

2.2.1. Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) aims to adapt models to downstream tasks with minimal parameter updates, achieving performance comparable to full fine-tuning under constrained computational and storage budgets. Existing PEFT methods generally fall into three categories. Adapter-based approaches insert lightweight trainable modules into intermediate layers of pre-trained models, enabling task adaptation while requiring only a small number of task-specific parameters. Prompt-based approaches introduce learnable prompt vectors at the input stage to guide models toward new tasks, preserving the original architecture but imposing additional input preprocessing requirements. Low-rank projection methods, represented by LoRA (Low-Rank Adaptation), decompose weight updates into low-rank factors, dramatically reducing the number of trainable parameters without altering model architecture or inference efficiency. Owing to its simplicity and robustness, LoRA has become a widely adopted PEFT baseline.

2.2.2. Variants of LoRA

Recent studies have explored improvements to LoRA in terms of expressive capacity, parameter allocation, and initialization. AdaLoRA [17] dynamically allocates ranks across different weights to improve parameter utilization and overall performance. DoRA [18] decomposes weight updates into magnitude and direction components and applies low-rank learning only to the directional component, thereby more closely approximating the behavior of full fine-tuning and enhancing training stability and generalization. These findings indicate that structural modifications alone do not fully bridge the performance gap between LoRA and full fine-tuning, and that initialization strategies play a critical role in determining LoRA performance.

2.2.3. LoRA Initialization and Decomposition-Based Methods

Recent work has emphasized the importance of structured initialization. PiSSA [19] constructs initial bases from principal singular vectors to better align with model structure. OLoRA [20] employs QR decomposition to form orthogonal subspaces, while CorDA [21] introduces contextual relevance into the initialization process. These approaches collectively demonstrate that carefully designed low-rank subspace initialization can significantly improve the optimization efficiency of PEFT.

Unlike decomposition-based methods, the proposed DWTLoRA adopts a signal-processing perspective and leverages the discrete wavelet transform to construct wavelet bases with explicit time–frequency locality. This design aligns LoRA initialization with speech structure by capturing both stationary components and transient variations in speech signals. As a result, the low-rank subspace acquires a stronger capacity to model speech dynamics from the outset, enhancing the effectiveness of LoRA in low-resource ASR tasks.

2.2.4. Module Integration

With the increasing adoption of large-scale speech models such as Whisper, integrating PEFT methods into these frameworks has become a growing trend. Building on the Whisper architecture, this work develops the Whisper-DWTLoRA model, in which wavelet-based structured initialization enables LoRA to achieve superior convergence behavior and training stability in low-resource ASR settings.

In contrast to conventional LoRA with random initialization, DWTLoRA aligns the adaptation subspace with the multi-scale structure of speech signals at the early stages of training. This alignment accelerates convergence, improves training stability, and preserves the original inference graph of Whisper without introducing additional computational overhead.

3. Wavelet-Structured Initialization

This section describes in detail how DWTLoRA introduces time–frequency structural priors of speech signals into LoRA through the discrete wavelet transform (DWT). The method aims to construct a low-rank subspace aligned with speech signal structure, thereby providing more informative optimization directions at the early stages of training.

DWTLoRA is built upon three core principles: (1) constructing a structured low-rank subspace using wavelet bases; (2) representing local variations in model parameters through multi-scale wavelet coefficients; and (3) optionally applying a residualization step that enables the adapter to focus on components of the weight matrix not explained by the imposed multi-scale structure.

3.1. Basic Formulation of LoRA

Let $W_0\in {\mathbb{R}}^{d\times k}$ denote the frozen pretrained weight matrix of a linear layer, where $d$ and $k$ are the output and input dimensions, respectively. LoRA parameterizes the trainable weight update $\mathsf{\Delta }W$ as a low-rank decomposition $\mathsf{\Delta }W = BA$, where $B \in {\mathbb{R}}^{d\times r}$ and $A \in {\mathbb{R}}^{r\times k}$ are learnable matrices with rank constraint $r \ll min(d,k)$, and $\alpha$ is a scaling factor controlling the magnitude of the update.

The effective weight used during forward propagation is therefore given by

$$W=W_0+{\displaystyle \frac{\alpha }{r}}BA$$

(1)

Under this formulation, gradients are propagated only through the low-rank factors $A$ and $B$. Applying the chain rule, the gradient of the loss function $L$ with respect to the effective weight update $\mathsf{\Delta }W$ is written as

$${\displaystyle \frac{\partial L}{\partial W}}={\displaystyle \frac{\alpha }{r}}{\displaystyle \frac{\partial L}{\partial \left(BA\right)}}$$

(2)

In standard LoRA implementations, the matrix $A$ is initialized to zero while $B$ is randomly initialized, which results in a near-zero effective update and thus weak gradient signals at the initial stage of training. This observation motivates the introduction of a structured initialization strategy for $B$.

3.2. Wavelet Transform and Structural Representation of Speech Signals

The discrete wavelet transform captures both temporal locality and multi-scale frequency characteristics, making it well suited for modeling non-stationary signals such as speech. Given a one-dimensional speech signal sequence $x \in {\mathbb{R}}^N$, its DWT can be expressed as

$$x=W_{wav}^{T}c$$

(3)

Here, $W_{wav}^T\in {\mathbb{R}}^{N\times N}$ denotes an orthogonal wavelet synthesis matrix determined by a chosen wavelet family, and $c$ is the corresponding vector of wavelet coefficients.

The coefficient vector $c$ consists of approximation coefficients $c_A^{\left(\mathrm{j}\right)}$ and detail coefficients $c_D^{\left(\mathrm{j}\right)}$ at different decomposition levels $j = 1,\dots ,J$, and can be written as

$$c={\left[c_A^{\left(1\right)},c_D^{\left(1\right)},c_D^{\left(2\right)},\dots ,c_D^{\left(J\right)}\right]}^T$$

(4)

This hierarchical organization enables simultaneous representation of global trends and fine-grained temporal details in speech signals.

From the wavelet basis vectors constituting $W_{wav}$, $r$ representative basis vectors ${\psi }_1, {\psi }_2, \dots , {\psi }_r$ are selected to form the matrix

$$W_w= \left[{\psi }_1, {\psi }_2, \dots , {\psi }_r\right]$$

(5)

Since the dimensionality $d$ of hidden representations is typically much larger than the original signal length $N$, each ${\psi }_i$ is mapped to the model dimension through interpolation or linear projection, yielding the initial basis matrix $B_0\in {\mathbb{R}}^{d\times r}$.

3.3. Wavelet-Based Initialization

The initial LoRA basis matrix is defined as

$$B_0= \left[{\psi }_1, {\psi }_2, \dots , {\psi }_r\right]$$

(6)

To ensure numerical stability and balanced scaling across columns, column-wise normalization is applied as

$${\tilde{B}}_0\left(:,i\right)={\displaystyle \frac{B_0\left(:,i\right)}{{‖B_0\left(:,i\right)‖}_2}}, i=1,\dots ,r$$

(7)

3.4. Residual-Aware Initialization

To account for components of the pretrained weight matrix $W_0$ that are not well represented in the wavelet subspace, a residual-aware initialization mechanism is introduced.

The projection of $W_0$ onto the subspace spanned by ${\tilde{B}}_0$ is computed as

$$P={\tilde{B}}_0^T{W}_0$$

(8)

The reconstructed component in the original parameter space is $W_{proj} = {\tilde{B}}_{0}P$ and the residual matrix is defined as

$$R=W_0-W_{proj}$$

(9)

which captures the portion of the pretrained weight that cannot be explained by the wavelet-structured low-rank subspace. Although $R$ is not directly optimized, its exclusion from the initial low-rank approximation allows the LoRA update to focus on complementary directions during subsequent training.

When residual-aware initialization is enabled, the LoRA factors are initialized as

$$B={\tilde{B}}_0,\hspace{1em} A=P$$

(10)

When residualization is disabled,

$$A=0,\hspace{1em} B={\tilde{B}}_0$$

(11)

In both cases, the resulting initial LoRA update is given by

$$\mathsf{\Delta }{W}_0= {\displaystyle \frac{\alpha }{r}}BA$$

(12)

where the rank parameter $r$ is identical to that in (1) and determines the dimensionality of the low-rank subspace.

3.5. Multi-Scale Fusion Mechanism

To further capture the multi-scale characteristics of speech signals, wavelet bases are selected from different decomposition levels and combined using learnable scale weights ${\lambda }_j$, enabling adaptive fusion of time–frequency features at different resolutions:

$${\tilde{B}}_0 = norm\left(\sum _{j=1}^J{\lambda }_j{\psi }_r^{\left(j\right)}\right)$$

(13)

Here, ${\psi }_r^{\left(j\right)}$ denotes the subset of wavelet basis vectors selected from the $j$-th decomposition level, and the normalization operator is identical to that defined in (7). During end-to-end training, the parameters ${\lambda }_j$ are jointly optimized with LoRA parameters, allowing the model to emphasize informative scales.

3.6. Initialization Algorithm

The complete DWTLoRA initialization procedure follows directly from Equations (6)–(13). Given the pretrained weight matrix $W_0$, target rank $r$, and a predefined wavelet family, wavelet basis vectors ${\psi }_r$ are first selected and mapped to the model dimension to construct $B_0$ as in (6).

If multi-scale fusion is enabled, bases from different decomposition levels are fused according to (13). The resulting matrix is normalized following (7) to obtain ${\tilde{B}}_0$.

If residual-aware initialization is enabled, the projection $P$ is computed using (8) and assigned to $A$ as in (10); otherwise, $A$ is initialized to zero as in (11). The final output consists of initialized matrices $B$ and $A$, which define the initial update direction $\mathsf{\Delta }{W}_0$ in (12).

This initialization strategy embeds prior knowledge of speech time–frequency structure into LoRA at the outset of training, thereby reducing reliance on random search during optimization and improving convergence efficiency and generalization performance under low-resource conditions.

4. Experiments

This section presents a systematic evaluation of DWTLoRA and compares Whisper + DWTLoRA with several LoRA-based methods. All experiments were conducted under identical hardware and training conditions using PyTorch 2.8.0. The operating system was Windows 11, with an AMD Ryzen 9 8940HX CPU, an NVIDIA GeForce RTX 5060 Laptop GPU, and 32 GB of system memory.

The training configuration followed a fixed setup: 15 training epochs, batch size of 4, the AdamW optimizer, an initial learning rate of 3 × 10⁻⁵, rank set to 32, alpha set to 64, and no bias parameters involved in training. Unless otherwise specified, all LoRA variants were applied only to the linear layers of the Whisper model.

4.1. Datasets

The experiments used a Southwest Mandarin speech corpus with a total duration of approximately 200 h. The dataset consists of conversational segments recorded by two independent speakers. Recording scenarios include predefined phrases, situational dialogues, and free-form conversations. Each utterance provides start and end timestamps, Simplified Chinese transcriptions, and anonymized speaker identifiers. The dataset was split into 160 h for training and 40 h for validation.

4.2. Evaluation Metrics

Model performance was primarily evaluated using the character error rate (CER), which measures the character-level discrepancy between recognition outputs and reference transcripts. The calculation relies on the Levenshtein edit distance by counting substitutions (S), deletions (D), and insertions (I), defined as:

$$CER =\left(S+D+I\right)/N$$

(14)

Here, $N$ denotes the total number of characters in the reference text. This metric provides an overall assessment of recognition accuracy.

The evaluation also employed the word error rate (WER), which follows the same definition as CER but performs the calculation at the word level rather than the character level.

Although CER is computed as the sum of substitutions, deletions, and insertions, this study focuses on the overall CER and does not separately optimize or analyze individual error types. The reported CER improvements therefore reflect the combined effect of changes in S, D, and I.

4.3. Experiment Results

Table 1 reports the ablation results of the individual components in DWTLoRA. The BASE configuration corresponds to the standard LoRA baseline without any additional modules. Configuration I introduces wavelet-based initialization on top of the BASE model. Configuration II further incorporates the multi-scale fusion mechanism in addition to wavelet-based initialization. Configuration III combines wavelet-based initialization with the residual strategy, while configuration IV represents the full DWTLoRA model, integrating wavelet-based initialization, multi-scale fusion, and the residual strategy.

Table 1. Ablation Results of DWTLoRA.

Name	Wavelet	Multi-Scale	Residual	CER
BASE	-	-	-	12.81%
I	√	-	-	12.62%
II	√	√	-	12.59%
III	√	-	√	12.56%
IV	√	√	√	12.52%

As shown in Table 1, introducing wavelet-based initialization alone reduced CER from 12.81% to 12.62%. Adding the multi-scale fusion mechanism further decreased CER to 12.59%, indicating that multi-scale structure independently improves performance. Enabling only the residual strategy yielded a CER of 12.56%, demonstrating that projection-based residual initialization encourages the adapter to focus on weight components not captured by the wavelet structure. In this work, CER is used as an overall evaluation metric that jointly reflects substitution, deletion, and insertion errors, rather than optimizing or analyzing each error type separately. Consequently, the observed improvements in CER should be interpreted as the combined effect across different error categories. While multi-scale fusion leads to only marginal gains in the final CER, its primary contribution lies in improving the robustness of LoRA initialization and providing more stable optimization directions during the early stages of training. Given the small magnitude of the CER differences, part of the improvement may be masked by statistical variability, and the benefit of multi-scale fusion is therefore more evident in training stability than in absolute recognition accuracy.

When all three mechanisms were enabled, DWTLoRA achieved the best CER of 12.52%, representing a clear improvement over the standard LoRA baseline at 12.81%. The overall training process remained stable and converged rapidly. These results demonstrate that DWTLoRA effectively exploits the multi-scale local structure of speech signals to achieve more stable training under low-resource conditions.

Figure 3 illustrates the training convergence curves of Whisper, Whisper + LoRA, and Whisper + LoRA + DWTLoRA. Short-term training results show that Whisper + LoRA slightly improved CER compared with the original Whisper model, while WER exhibited only marginal changes. After introducing structured initialization, Whisper + LoRA + DWTLoRA achieved a more pronounced reduction in CER, indicating that the proposed method uses limited training iterations more efficiently and enters an effective optimization regime earlier.

Figure 3. Training convergence curves of the three methods.

Table 2 compares CER and WER on the test set for the three prediction methods. As training progressed, Whisper + LoRA + DWTLoRA converged to a CER of 12.11% and a WER of 18.52% after 200 epochs. The introduction of DWTLoRA also reduced total training time to 5 h 39 min. These results confirm that Whisper + LoRA + DWTLoRA outperforms both the base Whisper model and Whisper + LoRA in speech recognition performance.

Table 2. Performance Comparison of Three Whisper-Based Methods.

Method	CER	WER	Training Time
Whisper	12.89%	18.96%	34 h 2 min
Whisper + LoRA	12.65%	18.76%	6 h 14 min
Whisper + LoRA + DWTLoRA	12.11%	18.52%	5 h 39 min

As shown in Figure 3, introducing LoRA in Whisper substantially reduces training loss, but yields only limited improvement in CER. This is because the loss reflects local probability optimization, whereas CER/WER are sequence-level metrics that are insensitive to small probability gains. In low-resource fine-tuning, LoRA mainly refines acoustic–language alignment and calibrates model confidence, with its benefits more evident in loss reduction than in final recognition results.

Figure 4 presents a comparative evaluation of DWTLoRA against several representative LoRA variants under identical training configurations on the Southwest Mandarin speech corpus. As shown, standard DoRA [18] achieves a CER of 12.80%, followed by AdaLoRA (12.68%) [17], OLoRA (12.67%) [20], and PiSSA (12.62%) [19]. Among all compared methods, DWTLoRA attains the lowest CER of 12.52%, demonstrating consistent performance gains over existing LoRA-based approaches. These results indicate that the proposed wavelet-based structured initialization provides more effective adaptation in low-resource ASR settings, even when compared with advanced LoRA variants that employ dynamic rank or parameter allocation strategies.

Figure 4. Comparison of CER among different LoRA-based methods on the Southwest Mandarin speech corpus (200 h) under identical training configurations.

5. Conclusions

This paper proposes DWTLoRA, a structured parameter-efficient fine-tuning method designed for low-resource ASR, and integrates it into the Whisper model. By leveraging the discrete wavelet transform, DWTLoRA constructs a low-rank adaptation subspace with multi-scale time–frequency locality, enabling LoRA to align with speech signal structure at the initialization stage. This alignment improves optimization efficiency and training stability during fine-tuning. The proposed approach preserves the original model architecture and introduces no additional inference overhead. Experiments on a 200-h Sichuan dialect corpus demonstrate that the Whisper-DWTLoRA framework outperforms standard LoRA and other PEFT baselines in both CER and WER, while achieving faster convergence and more stable training.

Future work will explore the application of DWTLoRA to multilingual and multimodal tasks, and investigate its combination with techniques such as dynamic rank allocation and task-adaptive initialization to further narrow the performance gap between PEFT and full fine-tuning. In addition, the wavelet-structured design of DWTLoRA shows promise for extension to larger-scale speech models and cross-modal modeling tasks, offering a more efficient and structured solution for low-resource scenarios.