Real-Time Pitch Estimation in Music Using Frequency Convolution Neural Network (FrCN) on Edge

Abstract

Accurate monophonic pitch estimation is central to music and speech systems, with practical applications often requiring small models that remain robust to noise and timbre variations. This paper proposes a Frequency Convolution Network (FrCN) for fundamental frequency and voicing estimation from Variable-Q Transform (VQT) features. The model applies dilated one-dimensional convolutions along the log-frequency axis, giving a large frequency receptive field with only 17.8k parameters. This design allows the model to compare harmonic evidence across frequency while keeping the parameter count low. The model is trained with spectral-domain noise augmentation and frequency scaling augmentation, which encourage noise invariance and pitch equivariance. Experiments on MDB-stem-synth, PTDB-TUG, and MIR-1K show that FrCN improves over a CNN baseline in clean and noisy conditions. On clean in-domain evaluation, FrCN reaches 98.8% RPA50 on MDB, 93.1% on PTDB, and 97.5% on MIR-1K. At 0 dB SNR, FrCN keeps RPA50 above 80% on all three in-domain validation sets. The model is also competitive with recent pitch estimation systems while using far fewer parameters. These results show that frequency axis dilation is effective for low-parameter, noise-robust pitch estimation.

1. Introduction

The fundamental frequency ($f_0$), which describes the perceived pitch of a voiced sound, is an important feature in music and speech analysis. Reliable $f_0$ estimation forms an important baseline for tasks such as singing analysis, tuning, transcription, audio-to-MIDI conversion, voice synthesis, timbre transfer, and pitch-conditioned audio effects [1,2,3,4,5,6]. These tasks require frame-level pitch estimation methods, which remain stable across different singers, instruments, pitch ranges, and acoustic conditions.

Classical pitch estimation algorithms use signal-processing techniques based on periodicity, autocorrelation, or spectral peak structure. The YIN algorithm uses a difference function to estimate pitch and is widely used for speech and music tasks [7]. pYIN extends this idea with a probabilistic model for threshold selection and temporal smoothing of the pitch [8]. The SWIPE algorithm estimates pitch by matching the spectrum to a sawtooth-inspired harmonic template highly relevant in music applications [9]. These methods are useful and interpretable; however, they underperform when the signal contains noise, accompaniment, or unusual timbres. Recent neural pitch estimators address this by learning to estimate the pitch directly from the audio data. CREPE showed that a convolutional network can estimate pitch directly from the waveform input with high accuracy [10]. Eventually, several models were proposed to improve the performance by introducing different standard signal processing biases. PESTO uses a VQT-based front-end along with a self-supervised transposition-equivariant objective, so that the model learns how pitch should move under frequency shifts [11]. PESTO-OT extends this idea with an optimal-transport formulation for translation-equivariant self-supervised learning [12]. YOLOPitch treats pitch estimation as a time–frequency detection problem, using a dual-branch YOLO-style model [13]. DJCM combines singing voice separation and vocal pitch estimation in a joint cascade model, which is useful when vocals and accompaniment are mixed together [14]. FCPE focuses on fast context-based pitch estimation [15], while SLASH studies self-supervised speech pitch estimation by correcting the signal processing based absolute-pitch estimation information [16]. Accordingly the pitch estimation has been addressed with different approaches like waveform models, log-frequency front-ends, detection-style architectures, joint source-separation pipelines, and self-supervised objectives.

A remaining challenge is the model parameter efficiency–accuracy tradeoff. Many pitch estimators are evaluated as stand-alone models, but often times they are only used as feature extraction modules within a larger pipeline, achieving the desired tasks. For example, a voice conversion or timbre transfer system may need pitch estimation, acoustic feature extraction, synthesis, and post-processing in the same real-time budget [2,17]. In this setting, efficient small pitch estimation models are invaluable for real-time systems, as a lightweight module leaves more computational capacity for the rest of the pipeline. Since pitch estimation is a frame level analysis, live applications such as vocal tuning and audio effects desire faster models. Smaller models are also advantageous for managing the computational budget of downstream tasks such as synthesis, sound effects, and transcription. In addition, there is a high demand for efficient low-parameter models for edge and embedded deployment in applications such as karaoke scoring systems, live translation, embedded music synthesizers, smart microphones, hearing aids, wearable voice monitors, etc. [18,19,20,21,22,23]. Jeong et al. [24] report CPU real-time factors of 0.56–0.59 on high-performance embedded devices even for efficient neural pitch-estimation networks, with larger baselines being much slower. This motivates the development of a pitch estimation model with a low parameter count and a low per-frame cost.

To address this gap, this study introduces a compact neural architecture for monophonic $f_0$ estimation. The model is designed to use a Variable-Q Transform (VQT) [25,26,27] front-end (discussed further in Section 2.3), which maps frequency bins onto a log-frequency axis. This representation is suitable for pitch estimation because pitch transposition manifests as a simple shift along this frequency axis. We propose a Frequency Convolution Network (FrCN) that applies dilated convolution along the frequency axis, allowing the model to capture a large frequency receptive field using minimal parameters. This design choice reflects the fact that accurate $f_0$ estimation involves the information captured from various harmonics across the frequency spectrum, not just detection of a single local spectral band [28].

The study also evaluates the robustness of the model performance from clean signals down to a 0 dB Signal-to-Noise Ratio (SNR), together with frequency scaling perturbations. This allows assessing the model performance for real life use, which often contains background noise and subtle frequency deviations. The role of noise augmentation for robustness to acoustic interference and frequency scaling augmentation for pitch-equivariant representations aligned with the log-frequency structure of VQT [11,12] was studied to optimize the model performance. The experiments were conducted using MDB-stem-synth [29], PTDB-TUG [30], and MIR-1K [31] datasets for pitch estimation, with CHiME-Home [32] and the MIR-1K accompaniment channel used as noise sources. The main contributions of this paper are:

• Introduction of a low-parameter FrCN architecture for VQT-based $f_0$ estimation that uses dilated frequency-axis convolution to capture the long-range harmonic structure.
• A detailed comparison between FrCN and CNN architectures at various matched parameter scales, to evaluate the importance of a larger receptive field in the frequency axis for accuracy and noise robustness.
• Training the models by combining spectral-domain noise along with frequency scaling augmentations and ablation studies to reveal the individual contributions on clean, noisy, and out-of-domain datasets.
• A benchmark evaluation against recent pitch-estimation models, demonstrating that the proposed 17.8k-parameter FrCN achieves competitive performance on MDB, PTDB, and MIR-1K despite its compact size.

2. Methods

This section describes the complete training and evaluation pipeline. The audio signal is first converted into a log-frequency VQT representation, where frame-level pitch alignment is done. Training uses octave balanced sampling together with noise and frequency scaling augmentations. The proposed Frequency Convolution Network (FrCN) then estimates both the pitch distribution and the voicing state for each VQT frame. See the Nomenclature section at the end of this article for detailed definitions of the variables used here.

2.1. Problem Formulation

Let $h_1^N$ denote a sampled sequence of an audio signal, $h\left(t\right)$, processed and sampled at a specific rate of $f_s$ Hz (e.g., 16 kHz), complying with the requirements of the WKS sampling theorem [33]. Each sample in $h_1^N$ may be denoted as $h_n:n\in \{1,\cdots ,N\}$. Variable-Q Transform (VQT) [27] is applied to the waveform, $h_1^N$, to produce a complex time–frequency spectrogram, $\mathbf{S}:{\mathcal{C}}^B\mapsto {\mathcal{R}}^R$, where R denotes the number of sequential frames of audio [33] and B represents the number of frequency bins. The goal is to formulate a multi-task problem where for each VQT frame, ${\mathbf{s}}_r$ (corresponding to the $r^{th}$ row of S, $r\in \{1,\dots ,R\}$), the following two measures are estimated: (a) voicing state ($v_r$) and (b) fundamental frequency ($f_{0r}$).

2.2. Datasets

Three primary audio datasets, MDB-stem-synth [29], PTDB-TUG [30], and MIR-1K [31], are used along with an environmental noise dataset, CHiME-Home [32], for training and evaluation. Details of these datasets are described below:

• MDB-stem-synth [29] consists of 230 resynthesized solo stems (musical excerpts) from MedleyDB, encompassing both instrument and vocal tracks. It provides precise fundamental frequency annotations instead of noisy manual labels, making it ideal for training a model on clean, monophonic pitch contours. This dataset is distributed at a sampling rate of $44.1$ kHz.
• PTDB-TUG [30] consists of 4718 audio recordings from 20 English speakers, encompassing both microphone and laryngograph, also known as Electroglottograph (EGG) [34] signals. It provides precise, objectively measured fundamental frequency annotations derived from the EGG data rather than subjective manual labels. This makes it ideal for training and evaluating models on clean, ground-truth pitch contours. This dataset is distributed at a sampling rate of 48 kHz.
• MIR-1K [31] is comprised of two-channel audio recordings. The two channels of the audio are processed as follows: (a) left channel containing the primary target audio, i.e., the isolated singing voice track, used for fundamental frequency estimation and (b) right channel containing the accompanying background music track. Because the right channel represents an acoustic disruption to the vocal line, it is isolated and repurposed to contribute directly as noise. This dataset is distributed at a sampling rate of 16 kHz.
• CHiME-Home [32] is comprised of approximately $6.8$ h of single channel audio recorded in various domestic settings. This dataset provides authentic household acoustic scenes, exposing the model to nonstationary, real-world background noise. This dataset is distributed at a sampling rate of 16 kHz and 48 kHz.

For consistency, the lowest sampling rate of 16 kHz is used across all datasets. For datasets with higher sampling rates, anti-aliasing and downsampling are performed as discussed in Section 2.3. Each dataset is split into $80\%$ for training and $20\%$ for testing at the track level, so that a recording appears in only one of the two parts. In-domain results train on the $80\%$ of a dataset and test on the $20\%$ of the same dataset. When a model is tested on a different dataset, the full $100\%$ of that dataset is used, since none of it is seen during training. Cross-dataset models are still trained on only the $80\%$ part of their training dataset.

2.3. Feature Extraction (VQT)

The input waveforms (including those belonging to noise datasets) are standardized by resampling to 16 kHz mono format, adhering to guidelines set forth by the WKS sampling theorem [33]. While high-fidelity audio is typically sampled at $44.1$ kHz or 48 kHz, we downsample to 16 kHz to optimize computational efficiency without sacrificing relevant spectral information. Because our subsequent time–frequency analysis is designed with a maximum center frequency of approximately $4.8$ kHz (the highest note on a piano is C8, which is 4186 Hz), a Nyquist frequency of 8 kHz safely captures the entire target frequency range while significantly reducing the temporal dimensionality and memory footprint for efficient model inference. For the MIR-1K dataset, the left and right channels are treated as individual mono channels, with the left and right channels being designated as MIR-1K and MIR-1K Noise respectively. This standardized signal is transformed into a time–frequency representation using the Variable-Q Transform (VQT), yielding a complex spectrogram S, $r\in \{1,\dots ,R\}$. The Variable-Q Transform, its log-frequency grid, bandwidth, quality factor, window length, and coefficients are described in Appendix A. This section only discusses how it is configured and applied within this work. The VQT is computed with a frequency resolution of $B_O=36$ bins per octave across $B=269$ frequency bins (starting at a minimum frequency $f_{\min }=27.5$ Hz, which is the A0 note on the piano, and reaching the aforementioned maximum of $f_{\max }=f_{\min }{2}^{(B-1)/B_O}\approx 4.8$ kHz) and a temporal frame shift, $T_r=10$ ms. The complex spectrogram is then sliced into discrete sequences of temporal frames $\{{\mathbf{s}}_1,\dots ,{\mathbf{s}}_R\}$ for processing, with ${\mathbf{s}}_r\in {\mathbb{C}}^B$. The additive bandwidth offset of Equation (A3) is set to $\eta =7$ Hz, and the coefficients are taken in magnitude form. Together with $f_s=16$ kHz, $T_r=10$ ms, $f_{\min }=27.5$ Hz, $B_O=36$ and $B=269$, these values are the complete front-end configuration.

2.4. Ground Truth Alignment

Since the datasets feature different annotation formats and resolutions, a synchronization pipeline is used to align the fundamental frequency and voicing targets with the temporal frames spaced with $T_r=10$ ms. Let ${\tau }_r=r·T_r$ be the center timestamp and $f_{0r}$ be the target frequency to be determined for the $r^{th}$ frame. Unvoiced frames are annotated with $f_{0r}=0$. MDB-stem-synth and MIR-1K provide explicit time–frequency pairs $({\tau }_a,f_{0a})$, where $f_{0a}=0$ denotes unvoiced frames. However, PTDB only provides a sequential list of frequencies $f_{0a}$ recorded at 10 ms shift and 32 ms window size. The missing timestamps ${\tau }_a$ are calculated using ${\tau }_a=(a·T_r)+16$ to properly recenter for the $a^{th}$ provided annotation.

Once the ground truth timestamps, ${\tau }_a$, are defined, ground-truth alignment is performed uniformly across all datasets using a tolerance-based interpolation strategy. If a frame center, ${\tau }_r$, falls within the tolerance window, $|{\tau }_r-{\tau }_a|\le 5$ ms, the target frequency, $f_{0r}$, will be determined by linear interpolation from adjacent $f_{0a}$ values. Although missing annotations were not observed in our datasets, we ensure robustness by marking any unannotated frame as unvoiced ($f_{0r}=0$). The voicing label $v_r$ for the $r^{th}$ frame is determined by

$$v_r=\mathbb{I}(f_{0r}>0)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{f}_{0r}>0\hfill \\ 0,\hfill & \mathrm{if}{f}_{0r}=0\hfill \end{array}\right.$$

(1)

This provides the training data as triplets $({\mathbf{s}}_r,f_{0r},v_r)$ for each frame in audio datasets and ${\mathbf{s}}_r$ for each frame in noise datasets.

2.5. Augmentation

The training pipeline uses two dynamic augmentations: additive noise augmentation and frequency scaling augmentation. These augmentations are sampled during training rather than fixed once during preprocessing, so the model observes different corrupted and shifted views of the same underlying frame across epochs. This increases the diversity of training examples without storing several augmented copies of the dataset.

Both augmentations are applied in the complex VQT domain. A direct time-domain implementation would require repeated waveform mixing, pitch scaling, and VQT recomputation, which is expensive because the VQT front-end has variable window lengths across the frequency domain. By applying augmentation after the VQT has been computed, the method keeps the training efficient while still being able to approximate the intended signal-level perturbations. Noise augmentation encourages invariance to perturbations unrelated to pitch, while frequency scaling augmentation encourages equivariance along the log-frequency axis.

2.5.1. Noise Augmentation

To improve the robustness of the system to realistic acoustic environments, the training pipeline is augmented with both structured harmonic interference (through MIR-1K Noise) and non-stationary ambient noise (through CHiME-Home). Due to the heavy front-end cost of the VQT computation for preprocessing, standard waveform level data augmentation is avoided, to facilitate dynamic data augmentation across epochs. Instead, data augmentation is performed in the complex spectral domain which serves as an approximation to direct time-domain application of noise.

Let ${\mathbf{s}}_{\mathrm{c}}\in {\mathbb{C}}^B$ and ${\mathbf{s}}_{\mathrm{n}}\in {\mathbb{C}}^B$ denote VQT frames from the clean and noisy audio respectively. Let us define the average power of a complex spectral frame as

$$P\left(\mathbf{s}\right)\triangleq {\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{\left|s_b\right|}^2$$

(2)

For variability in the Signal-to-Noise Ratio (SNR), a baseline scaling factor ${\beta }_{\mathrm{orig}}$ is sampled from a uniform distribution, ${\beta }_{\mathrm{orig}}\sim \mathcal{U}({\beta }_{\min },{\beta }_{\max })$, where ${\beta }_{\min }=0.0$ and ${\beta }_{\max }=0.5$. However, ${\beta }_{\mathrm{orig}}$ cannot be directly used for mixing due to the inherent discrepancy of absolute signal energy across the various datasets. To address this, the corrected scaling factor ${\beta }_{\mathrm{corr}}$ is calculated by normalizing the noise amplitude relative to the signal as follows:

$${\beta }_{\mathrm{corr}}={\beta }_{\mathrm{orig}}·\sqrt{{\displaystyle \frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{P\left({\mathbf{s}}_{\mathrm{n}}\right)+ϵ}}}$$

(3)

where $ϵ={10}^{-6}$ ensures numerical stability. The augmented frame ${\mathbf{s}}_{\mathrm{mix}}$ is then given as

$${\mathbf{s}}_{\mathrm{mix}}={\mathbf{s}}_{\mathrm{c}}+{\beta }_{\mathrm{corr}}·{\mathbf{s}}_{\mathrm{n}}$$

(4)

This formulation allows a deterministic correlation between ${\beta }_{\mathrm{orig}}$ and the frame-level SNR in decibels ${\mathrm{SNR}}_{\mathrm{dB}}$. Substituting ${\beta }_{\mathrm{corr}}$ and simplifying,

$${\mathrm{SNR}}_{\mathrm{dB}}=10{\log }_{10}\left({\displaystyle \frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{P({\beta }_{\mathrm{corr}}·{\mathbf{s}}_{\mathrm{n}})}}\right)=10{\log }_{10}\left({\displaystyle \frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{{\beta }_{\mathrm{corr}}^2·P\left({\mathbf{s}}_{\mathrm{n}}\right)}}\right)$$

(5)

Replacing ${\beta }_{\mathrm{corr}}$, the fraction inside the logarithm reduces as follows:

$${\displaystyle \frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{{\beta }_{\mathrm{corr}}^2·P\left({\mathbf{s}}_{\mathrm{n}}\right)}}={\displaystyle \frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{{\left({\beta }_{\mathrm{orig}}·\sqrt{\frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{P\left({\mathbf{s}}_{\mathrm{n}}\right)+ϵ}}\right)}^2·P\left({\mathbf{s}}_{\mathrm{n}}\right)}}={\displaystyle \frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{{\beta }_{\mathrm{orig}}^2·\frac{P\left({\mathbf{s}}_{\mathrm{c}}\right)}{P\left({\mathbf{s}}_{\mathrm{n}}\right)+ϵ}·P\left({\mathbf{s}}_{\mathrm{n}}\right)}}={\displaystyle \frac{1}{{\beta }_{\mathrm{orig}}^2}}\left(1+{\displaystyle \frac{ϵ}{P\left({\mathbf{s}}_{\mathrm{n}}\right)}}\right)$$

(6)

Since $ϵ={10}^{-6}$ and assuming the noise power $P\left({\mathbf{s}}_{\mathrm{n}}\right)\gg ϵ$ for any non-silent noise frame, the term ${\displaystyle \frac{ϵ}{P\left({\mathbf{s}}_{\mathrm{n}}\right)}}$ becomes negligible, yielding $\left(1+{\displaystyle \frac{ϵ}{P\left({\mathbf{s}}_{\mathrm{n}}\right)}}\right)\approx 1$. Therefore, the theoretical SNR simplifies to

$${\mathrm{SNR}}_{\mathrm{dB}}\approx 10{\log }_{10}\left({\displaystyle \frac{1}{{\beta }_{\mathrm{orig}}^2}}\right)=-20{\log }_{10}\left({\beta }_{\mathrm{orig}}\right)$$

(7)

Conversely, in the trivial case where the noise frame is silent or near-silent ($P\left({\mathbf{s}}_{\mathrm{n}}\right)\lesssim ϵ$), the stability term $ϵ$ dominates the denominator and bounds the scaling factor. This prevents the algorithm from excessively amplifying near-silent noise, ensuring that the target ${\mathrm{SNR}}_{\mathrm{dB}}$ formulation safely breaks down to a clean signal reconstruction, ${\mathbf{s}}_{\mathrm{mix}}\approx {\mathbf{s}}_{\mathrm{c}}$.

As shown in Figure 1, for the current parameters, the median divides the samples into 50% medium to high-noise (6–12 dB) and 50% low noise (12 dB to near-clean) regions. This balance exposes the model to medium-to-high noise environments during training while observing cleaner examples, encouraging robustness without forcing overfitting to heavily corrupted inputs. This also provides an interpretable way to control augmentation severity: the upper limit ${\beta }_{\max }$ defines the minimum SNR floor, while ${\beta }_{\min }$ controls the highest-SNR tail.

Figure 1. Noise scaling distribution used for spectral-domain augmentation. The (left) panel shows the uniformly sampled baseline factor ${\beta }_{\mathrm{orig}}\sim \mathcal{U}({\beta }_{\min },{\beta }_{\max })$. The (right) panel shows the induced frame-level ${\mathrm{SNR}}_{\mathrm{dB}}$ obtained from ${\mathrm{SNR}}_{\mathrm{dB}}\approx -20{\log }_{10}\left({\beta }_{\mathrm{orig}}\right)$. The logarithmic mapping places more augmented frames in the medium-to-high noise region while retaining a tail of cleaner examples.

To efficiently draw the noise frames, both noise datasets are converted to the complex VQT domain once before training, and their frames are pooled into a single noise bank. At every training step, one noise frame is drawn at random from this bank, for each sample in the mini-batch. So the two datasets are represented in proportion to their frame counts. The noise and frequency scaling augmentations are applied to every training sample in each epoch, with ${\beta }_{\mathrm{orig}}$ and the shift drawn again for each sample. There is no separate probability of applying an augmentation since ${\beta }_{\mathrm{orig}}=0$ is already part of the sampling range.

2.5.2. Frequency Scaling Augmentation

Since the VQT representation is computed along a logarithmic frequency axis, a translation along this axis corresponds directly to a multiplicative scaling of the frequency, i.e., a pitch transposition. Small shifts along the VQT frequency axis therefore constitute a natural augmentation: they simulate pitch drift or slight intonation variation without altering the timbral structure of the frame. To implement this, we apply a zero-padded shift of small magnitude along the VQT frequency axis, ensuring the fundamental frequency remains within the analysis range and the harmonic structure is preserved.

Let ${\mathbf{s}}_{\mathrm{shift}}\in {\mathbb{C}}^B$ denote the shifted frame obtained by translating ${\mathbf{s}}_{\mathrm{c}}$ by k bins along the VQT frequency axis with zero-padding:

$$s_{\mathrm{shift},b}=\left\{\begin{array}{cc}{s}_{\mathrm{c},b-k},\hfill & \mathrm{if}1\le b-k\le B\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

where $b\in \{1,\dots ,B\}$ is the VQT frequency-bin index. In our analysis, the shift k is drawn uniformly over the integers in $[-27,+36]$ bins. At 36 bins per octave, i.e., 3 bins per semitone, this covers $-9$ to $+12$ semitones. If ${\mathbf{s}}_{\mathrm{c}}$ is a voiced frame whose fundamental frequency is $f_{0r}$ Hz, then its pitch in cents is defined as

$${¢}_r=1200{\log }_2\left(f_{0r}\right)$$

(9)

When a shift of k bins is applied and the VQT uses $B_O$ bins per octave, the equivalent pitch displacement in cents is

$$\Delta ¢={\displaystyle \frac{1200k}{B_O}}$$

(10)

resulting in the shifted target pitch

$${¢}_{r,\mathrm{shift}}={¢}_r+\Delta ¢$$

(11)

and the corresponding shifted target frequency

$$f_{0r,\mathrm{shift}}=2^{{\displaystyle \frac{{¢}_{r,\mathrm{shift}}}{1200}}}$$

(12)

Each voiced frame thus yields two input–target pairs, $({\mathbf{s}}_{\mathrm{c}},{¢}_r)$ and $({\mathbf{s}}_{\mathrm{shift}},{¢}_{r,\mathrm{shift}})$, which are appended to the training set. Combined with additive noise augmentation, this gives rise to four views per training frame.

2.6. Proposed Architecture

The primary objective is fundamental frequency prediction invariant to timbre variations, instrument source, and absolute pitch transposition. The end-to-end pipeline of the proposed architecture is illustrated in Figure 2.

Figure 2. End-to-end FrCN pipeline for frame-level pitch and voicing estimation. The audio waveform is converted to a VQT spectrogram, aligned with frame-level pitch and voicing labels, augmented in the spectral domain during training, and passed through the FrCN model to produce a pitch distribution and a voicing estimate.

Time Delay Neural Networks (TDNNs) [35,36] are widely used in sequence modeling tasks, such as speech and audio processing. The primary advantage of TDNN is the use of dilated convolution which allows the network to expand its field of view (receptive field) without increasing computational weight. Unlike traditional CNNs which only analyze adjacent data points, a dilated kernel skips a fixed number of data points in between, increasing the receptive field of the network.

In the proposed architecture, this 1-D dilation mechanism is applied along the frequency axis allowing the network to capture long-range harmonic relationships across a vast frequency span while using fewer weights. Because the input features are obtained through log-frequency VQT, harmonic intervals maintain a fixed geometric distance along the frequency axis irrespective of the absolute pitch. The sliding dilated kernel can therefore identify these invariant relationships efficiently, enabling low-parameter estimation. To avoid confusion with time-domain sequence models, this new architecture which operates in the frequency domain is henceforth referred to as a Frequency Convolution Network (FrCN)—for contrast see the detailed description in [36] for the time-domain version.

2.6.1. Signal Preprocessing

The preprocessing module is shown in Figure 3. The input to the model is the complex spectrogram frame, ${\mathbf{s}}_r\in {\mathbb{C}}^B$, obtained after noise augmentation during training and the clean spectral frame during inference. The magnitude representation ${\mathbf{x}}_r=\left|{\mathbf{s}}_r\right|\in {\mathbb{R}}^B$ is computed to map it into a real-valued feature space.

Figure 3. Signal preprocessing module used before the FrCN blocks. Each complex VQT frame is converted to magnitude form and represented through four views: identity, learnable logarithmic compression, area normalization, and power normalization. The four views are stacked as input channels for the front-end adaptation layer.

Instead of passing this directly into the model, a deterministic non-linear transformation is applied to expose different structural attributes of the signal. Four views are computed as follows:

• Identity View: Unmodified spectral frame to preserve absolute spectral peaks.
• Learnable Logarithmic View: Compresses broad dynamic-range and enhances the low-energy harmonic partials into a more visible region. The learnable scaling is determined by the parameter $\alpha$.

  $${\mathbf{x}}_{r,\log }=\ln \left(1+\ln (1+e^{\alpha }){\mathbf{x}}_r\right)$$

  (13)
• Area-Normalized View: Normalizes the frame by its total energy, making the representation invariant to global loudness and interpretable as a probability mass function over frequency bins.

  $${\mathbf{x}}_{r,\mathrm{area}}={\displaystyle \frac{{\mathbf{x}}_r}{{\sum }_{b=1}^B{x}_{rb}+ϵ}}$$

  (14)
• Power-Normalized View: Scales by the RMS energy of the frame, providing a view that is stable with respect to the second-order signal power.

  $${\mathbf{x}}_{r,\mathrm{pow}}={\displaystyle \frac{{\mathbf{x}}_r}{\sqrt{\frac{1}{B}{\sum }_{b=1}^B{x}_{rb}^2}+ϵ}}$$

  (15)

These four views are stacked in the channel dimension to get the preprocessed input ${\mathbf{X}}_r\in {\mathbb{R}}^{4\times B}$, giving the model a richer and more structured representation of the spectral frame to work with.

2.6.2. Front-End Adaptation Layer

The processed multi-view tensor $X_r$ is passed through a front-end adaptation block that increases the model latent space from 4 to H. $H=18$ was chosen since it was the smallest dimension giving consistent results. This is performed by using 1D convolution with a kernel size of 3 and a padding of 1, which ensures length preservation. Sigmoid Linear Unit (SiLU) activation was used to introduce non-linearity, generating ${\mathbf{H}}_{r,0}\in {\mathbb{R}}^{H\times B}$. H will act as the hidden latent dimension the model moving forward.

2.6.3. FrCN Feature Extraction

The FrCN block structure is shown in Figure 4. The core of the model is a stack of L FrCN blocks. Each block consists of dilated convolution followed by SiLU non-linearity and batch normalization. To increase the receptive field of the network with depth, the dilation at layer $l\in \{1,2,\dots ,L\}$ is given by $d_l=2^{\max (0,l-2)}$. The convolution layer in all blocks uses a fixed kernel of size 5 with padding set dynamically based on the layer dilation, $p_l=2d_l$, to preserve the sequence length. Scaled residual connections are used after the block computation, given by

$${\mathbf{H}}_{r,l}=\mathrm{BN}\left(\mathrm{SiLU}\left(\mathrm{Conv}1\mathrm{D}\left({\mathbf{H}}_{r,l-1},d_l,p_l\right)\right)+\gamma {\mathbf{H}}_{r,l-1}\right)$$

(16)

where BN is the Batch Normalization, SiLU is the SiLU activation function and Conv1D is the 1D convolution operation with dilation $d_l$ and padding $p_l$. The scaling factor, $\gamma =0.66<1$, is used to attenuate the residual contribution, to stabilize the gradient flow and prevent the model from heavily depending on identity connections in the deeper layers.

Figure 4. FrCN feature-extraction block. A stack of dilated one-dimensional convolutions is applied along the VQT frequency axis. The dilation schedule expands the receptive field over frequency without a large increase in parameters, allowing the model to combine harmonic evidence from distant frequency bins.

2.6.4. Output Heads

The output head structure is shown in Figure 5. The obtained feature map after feature extraction, ${\mathbf{H}}_{r,L}\in {\mathbb{R}}^{H\times B}$, is transposed to obtain ${\mathbf{H}}_{r,\mathrm{d}}\in {\mathbb{R}}^{B\times H}$, to treat each frequency bin independently of others. Two parallel output heads are used for frequency and voicing estimation.

Figure 5. Output heads for pitch and voicing estimation. The final FrCN feature map is transposed so that each frequency bin is processed by shared MLP heads. The pitch head produces a probability distribution over pitch bins, while the voicing head aggregates bin-level information into a single voiced/unvoiced decision. Note that sigmoid is the bias-free point-wise activation function.

For frequency estimation, each bin’s feature vector of dimension H is passed through a two-layer MLP (of structure $H\to H\to 1$) with SiLU activation between the layers. This generates a scalar pitch estimate for each bin which is passed through a softmax layer to obtain the probability distribution over the bins.

For voicing estimation, similar to frequency estimation, a two-layer MLP (of structure $H\to H\to 1$) with SiLU activation between the layers is used. Following this, a linear layer aggregates the bin-level reference into a single voicing scalar which is passed through a sigmoid to obtain the final voicing prediction.

2.7. Training Procedure

There are two primary challenges in frame-level voicing and $f_0$ estimation: (a) dataset imbalance due to an unequal number of voiced and unvoiced frames as well as non-uniform distribution of pitch in voiced frames and (b) robustness to realistic audio perturbations like high noise or small frequency axis drifts.

2.7.1. Balanced Mini-Batch Sampling

To address dataset imbalance, we propose a balanced mini-batch sampling strategy for each epoch, which balances the number of voiced and unvoiced frames as well as octave index uniformity among the voiced frames. The voiced frames in the datasets are organized into octave groups using the frame’s octave index $o_r$, defined as follows.

The cent is the logarithmic unit used to measure musical intervals, defined exactly as $1/100$ of a semitone (half-step). To quantify frequencies as intervals a reference frequency, $f_{\mathrm{ref}}$, must be defined. The value of any frequency f Hz in terms of an interval in cents is defined as

$$¢=1200{\log }_2{\displaystyle \frac{f}{f_{\mathrm{ref}}}}$$

where $f_{\mathrm{ref}}$ is a reference frequency. For simplicity of computation, $f_{\mathrm{ref}}=1$ Hz is used in this entire work. Let $f_{\min }$ and ${¢}_{\min }=1200{\log }_2\left(f_{\min }\right)$ be the minimum frequency and its corresponding cent value used for VQT (see Section 2.3). For a voiced frame with target frequency $f_{0r}$, its pitch in cents is

$${¢}_r=1200{\log }_2\left(f_{0r}\right)$$

(17)

and its octave index $o_r$ is given by

$$o_r=⌊{\displaystyle \frac{{¢}_r-{¢}_{\min }}{1200}}⌋$$

(18)

During each epoch, the effective epoch size is controlled by separate voiced and unvoiced sample caps, $N_{\mathrm{epoch}}=N_{\mathrm{voiced}}+N_{\mathrm{unvoiced}}$, where $N_{\mathrm{voiced}}$ and $N_{\mathrm{unvoiced}}$ denote the number of voiced and unvoiced frames sampled per epoch, respectively. Groups whose sample count falls below a minimum fraction of the largest group are merged with their smaller neighbor to prevent very small octave bins from being heavily oversampled. The voiced samples are then drawn uniformly from the octave groups, whereas unvoiced samples are drawn randomly from the available unvoiced frames. This ensures that the model learns from a controlled uniform subset of the data at every epoch, optimizing the training and preventing overfitting to the most frequent pitch regions.

2.7.2. Multi-View Augmentation

For robustness to real-world acoustic conditions, we employ the multi-view augmentation strategy described in Section 2.5. The four input views are: the original clean spectral frame ${\mathbf{x}}_r^{\mathrm{clean}}$, a noise-augmented version ${\mathbf{x}}_r^{\mathrm{noisy}}$, a VQT shifted version ${\mathbf{x}}_r^{\mathrm{shift}}$, and a combined VQT shift-and-noise version ${\mathbf{x}}_r^{\mathrm{shift}+\mathrm{noisy}}$.

During training, the model processes all four views for each mini-batch. The clean and noisy views use the original $f_0$ target, while the VQT shifted and VQT shift-plus-noise views use the shifted $f_0$ target for supervision. This ensures that the model learns invariance to noise augmentation and equivariance to frequency scaling. The pitch-estimation loss is computed only for voiced frames since unvoiced frames do not contain a meaningful frequency target. This multi-view training strategy therefore jointly improves pitch accuracy, voicing reliability, and robustness under realistic acoustic perturbations.

2.7.3. Pitch Target and Loss

For voiced frames, the scalar pitch target is converted into a soft distribution over the discrete output pitch axis. Let $\mathbf{a}=[a_1,a_2,\dots ,a_B]$ denote the pitch in cents for each of the B output pitch bins.

For a voiced frame with the target frequency $f_{0r}$, the target pitch in cents is ${¢}_r=1200{\log }_2\left(f_{0r}\right)$ (reference frequency of 1 Hz for all cent calculations). We define a smooth, bell-shaped discrete target probability mass function (PMF) for use in the loss function, which allows small pitch errors while penalizing severe errors. The target probability at pitch $a_b$ as a function of the target ${¢}_r$ is defined as

$$q_{rb}=p(a_b;{¢}_r)\triangleq {\displaystyle \frac{\exp \left[-\frac{1}{2}{\left(\frac{a_b-{¢}_r}{{\sigma }_{¢}}\right)}^2\right]}{{\sum }_{j=1}^B\exp \left[-\frac{1}{2}{\left(\frac{a_j-{¢}_r}{{\sigma }_{¢}}\right)}^2\right]}}$$

(19)

where ${\sigma }_{¢}=35$ cents dictates the spread of the PMF across the B pitch bins. For a standard voiced frame with target ${¢}_r$, the target PMF vector, ${\mathbf{q}}_r\in {\mathbb{R}}^B$, has elements, $q_{rb}=p(a_b;{¢}_r)$. Similarly, for VQT shifted views, the shifted target PMF vector ${\mathbf{q}}_r^{\mathrm{shift}}\in {\mathbb{R}}^B$ has elements $q_{rb}^{\mathrm{shift}}=p(a_b;{¢}_{r,\mathrm{shift}})$.

Let $m\in \{\mathrm{clean},\mathrm{noisy},\mathrm{shift},\mathrm{shift}+\mathrm{noisy}\}$ denote a specific view. For a given frame, let ${\mathbf{z}}_r^m=[z_{r1}^m,\dots ,z_{rB}^m]$ be the pitch logits predicted by the model. These are converted into a predicted pitch distribution ${\widehat{\mathbf{p}}}_r^m$, using the softmax function:

$${\widehat{p}}_{rb}^m={\displaystyle \frac{\exp \left(z_{rb}^m\right)}{{\sum }_{j=1}^B\exp \left(z_{rj}^m\right)}}$$

(20)

For a voiced frame, the pitch loss is computed using the Kullback–Leibler (KL) divergence from the target PMF to the predicted distribution:

$${\mathcal{L}}_{\mathrm{p}}\left({\widehat{\mathbf{p}}}_r^m,{\mathbf{q}}_r\right)=D_{\mathrm{KL}}\left({\mathbf{q}}_r\parallel {\widehat{\mathbf{p}}}_r^m\right)={\displaystyle \sum _{b=1}^B}{q}_{rb}\ln \left({\displaystyle \frac{q_{rb}}{{\widehat{p}}_{rb}^m}}\right)$$

(21)

This pitch loss is aggregated across all four training views. To ensure the pitch branch is invariant to noise but equivariant to VQT frequency-axis shifts, the clean and noisy views are evaluated against the original target, ${\mathbf{q}}_r$, while the shifted views are evaluated against the shifted target, ${\mathbf{q}}_r^{\mathrm{shift}}$:

$${\mathcal{L}}_{\mathrm{dist},r}={\mathcal{L}}_{\mathrm{p}}\left({\widehat{\mathbf{p}}}_r^{\mathrm{clean}},{\mathbf{q}}_r\right)+{\mathcal{L}}_{\mathrm{p}}\left({\widehat{\mathbf{p}}}_r^{\mathrm{noisy}},{\mathbf{q}}_r\right)+{\mathcal{L}}_{\mathrm{p}}\left({\widehat{\mathbf{p}}}_r^{\mathrm{shift}},{\mathbf{q}}_r^{\mathrm{shift}}\right)+{\mathcal{L}}_{\mathrm{p}}\left({\widehat{\mathbf{p}}}_r^{\mathrm{shift}+\mathrm{noisy}},{\mathbf{q}}_r^{\mathrm{shift}}\right)$$

(22)

2.7.4. Consistency Regularization

In addition to the supervised pitch losses, consistency regularization is applied across the augmented views. The clean prediction ${\widehat{\mathbf{p}}}_r^{\mathrm{clean}}$ is used as the reference distribution and is detached during this consistency computation, so it acts as a stable teacher for the augmented views.

For the noisy view, the prediction is encouraged to remain close to the clean prediction:

$${\mathcal{L}}_{\mathrm{cons},\mathrm{noisy},r}=D_{\mathrm{KL}}\left({\widehat{\mathbf{p}}}_r^{\mathrm{clean}}\parallel {\widehat{\mathbf{p}}}_r^{\mathrm{noisy}}\right)$$

(23)

For the shifted views, the clean prediction is first shifted along the output pitch axis using the zero-padded shift operator ${\mathcal{S}}_k(·)$ (see Section 2.5 for k):

$${\tilde{\mathbf{p}}}_r^{\mathrm{shift}}={\mathcal{S}}_k\left({\widehat{\mathbf{p}}}_r^{\mathrm{clean}}\right)$$

(24)

The shifted consistency losses are then

$${\mathcal{L}}_{\mathrm{cons},\mathrm{shift},r}=D_{\mathrm{KL}}\left({\tilde{\mathbf{p}}}_r^{\mathrm{shift}}\parallel {\widehat{\mathbf{p}}}_r^{\mathrm{shift}}\right)$$

(25)

$${\mathcal{L}}_{\mathrm{cons},\mathrm{shift}+\mathrm{noisy},r}=D_{\mathrm{KL}}\left({\tilde{\mathbf{p}}}_r^{\mathrm{shift}}\parallel {\widehat{\mathbf{p}}}_r^{\mathrm{shift}+\mathrm{noisy}}\right)$$

(26)

These individual consistency losses are weighted to obtain total consistency loss given by

$${\mathcal{L}}_{\mathrm{consistency},r}={\lambda }_{\mathrm{noisy}}{\mathcal{L}}_{\mathrm{cons},\mathrm{noisy},r}+{\lambda }_{\mathrm{shift}}{\mathcal{L}}_{\mathrm{cons},\mathrm{shift},r}+{\lambda }_{\mathrm{shift}+\mathrm{noisy}}{\mathcal{L}}_{\mathrm{cons},\mathrm{shift}+\mathrm{noisy},r}$$

(27)

The loss weights are set to ${\lambda }_{\mathrm{noisy}}=0.05$, ${\lambda }_{\mathrm{shift}}=0.05$, and ${\lambda }_{\mathrm{shift}+\mathrm{noisy}}=0.025$.

2.7.5. Voicing Loss

The voicing branch is trained using binary cross-entropy with logits. Let $v_r\in \{0,1\}$ be the ground-truth voicing label (see Section 2.4) and $u_r$ be the predicted voicing logit. The voicing loss is

$${\mathcal{L}}_{\mathrm{voice},r}=-\left[v_r\ln \sigma \left(u_r\right)+(1-v_r)\ln \left(1-\sigma \left(u_r\right)\right)\right]$$

(28)

where $\sigma (·)$ is the sigmoid function. Unlike the pitch loss, the voicing loss is computed for both voiced and unvoiced frames.

2.7.6. Total Loss

For every mini-batch, the total loss is obtained by averaging the frame-level losses. Let $\mathcal{B}$ denote the set of frame indices in a mini-batch, and let $\mathcal{V}\subseteq \mathcal{B}$ denote the set of voiced frame indices in the mini-batch:

$$\mathcal{V}=\{r\in \mathcal{B}:v_r=1\}$$

(29)

The pitch loss is computed only over voiced frames:

$${\mathcal{L}}_{\mathrm{pitch}}={\displaystyle \frac{1}{\left|\mathcal{V}\right|}}\sum _{r\in \mathcal{V}}\left[{\mathcal{L}}_{\mathrm{dist},r}+{\mathcal{L}}_{\mathrm{consistency},r}\right]$$

(30)

The voicing loss is computed over all frames in the mini-batch:

$${\mathcal{L}}_{\mathrm{voicing}}={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{r\in \mathcal{B}}{\mathcal{L}}_{\mathrm{voice},r}$$

(31)

The final training objective is

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{pitch}}+{\mathcal{L}}_{\mathrm{voicing}}$$

(32)

where $\left|\mathcal{V}\right|$ is the number of voiced frames in the mini-batch and $\left|\mathcal{B}\right|$ is the total number of frames in the mini-batch.

2.8. Implementation Details

2.8.1. Architecture Specification and the CNN Baseline

Most real-time pitch estimators, proposed recently, are built on convolutional networks with local kernels, so a standard CNN is the natural baseline for the proposed model. A CNN baseline that matches the FrCN in every respect other than dilation was defined, by setting the dilation to 1 in every block of the FrCN. The preprocessing, adaptation layer, kernel size, channel width H, normalization, activations, residual scaling, output heads and depth L are the same in both models. The padding was defined as $p_l=2d_l$ to ensure the length of the frequency axis is preserved. The full layer specification is given in Table 1. The dilation of the FrCN is $d_l=2^{\max (0,l-2)}$ and that of the CNN baseline is $d_l=1$, so the dilation schedule is the only thing that distinguishes them.

Table 1. Layer specification of the proposed FrCN and the CNN baseline at $L=10$ and $H=18$, for a VQT frame of $B=269$ bins. The two architectures are identical except for the dilation column.

Stage	In	Out	Kernel Size	Stride	Bias	Dilation ${\mathit{d}}_{\mathit{l}}$/Padding ${\mathit{p}}_{\mathit{l}}$
						FrCN	CNN
Preprocessing (4 views)	1	4	—	—	—	—	—
Adaptation Conv1D + SiLU	4	18	3	1	yes	1/1	1/1
Block $l=1$	18	18	5	1	no	1/2	1/2
Block $l=2$	18	18	5	1	no	1/2	1/2
Block $l=3$	18	18	5	1	no	2/4	1/2
Block $l=4$	18	18	5	1	no	4/8	1/2
Block $l=5$	18	18	5	1	no	8/16	1/2
Block $l=6$	18	18	5	1	no	16/32	1/2
Block $l=7$	18	18	5	1	no	32/64	1/2
Block $l=8$	18	18	5	1	no	64/128	1/2
Block $l=9$	18	18	5	1	no	128/256	1/2
Block $l=10$	18	18	5	1	no	256/512	1/2
Pitch head (MLP + SiLU)	18	1	—	—	yes	—	—
Voicing head (MLP + SiLU)	18	1	—	—	yes	—	—
Voicing aggregation (linear)	269	1	—	—	yes	—	—

Every block applies dilated convolution, SiLU, batch normalization, and the scaled residual bypass of Equation (16). Both models have the same parameter count, given by

$$N_{\mathrm{params}}=1227+1656L,$$

resulting in 7851 parameters at $L=4$ and $17,787$ at $L=10$.

2.8.2. Computational Parity and Receptive Field

Dilation only changes which input positions that a kernel reads but not how many multiply-accumulate (MAC) operations it performs. Hence, the two models have the same computational cost of around $4.60$ MMAC per frame in the largest configuration given in Table 1.

On the other hand, dilation affects the receptive field by increasing it after each step. For kernel size k and dilation $d_l$, each block extends the receptive field by $(k-1)d_l$ bins. Hence, for an L layer model, the total receptive field would be given by $1+(3-1)+{\sum }_{l=1}^L(k-1)d_l$ bins, where 1 and $(3-1)$ are artifacts from the preprocessing convolutions. At $L=10$ this results in 2051 bins for the FrCN and 43 bins for the CNN baseline. The VQT frame is only 269 bins wide, or $7.47$ octaves, so the FrCN reaches across the entire analyzed band while the CNN baseline sees only 43 bins, about $1.19$ octaves. A span of $1.19$ octaves is too narrow to link a fundamental to its upper harmonics, which is what the architecture is meant to do. Since the two models are matched in parameter count and in computational complexity, the comparison in Section 3 only changes the receptive field, while keeping everything else fixed. Table 2 lists the receptive field at the five depths used in Section 3.2.

Table 2. Frequency receptive field of the FrCN and the CNN baseline at each matched parameter scale, in VQT bins and in octaves at $B_O=36$ bins per octave. The analyzed band is 269 bins, or $7.47$ octaves.

Layers	Parameters	FrCN		CNN Baseline
		Bins	Octaves	Bins	Octaves
4	7.9k	35	0.97	19	0.53
6	11.2k	131	3.64	27	0.75
7	12.8k	259	7.19	31	0.86
8	14.5k	515	14.31	35	0.97
10	17.8k	2051	56.97	43	1.19

2.8.3. Selection of Fixed Hyperparameters

Four values are kept fixed in all experiments. Three of them follow from the VQT settings.

• Target spread ${\sigma }_{¢}=35$ cents (Equation (19)). The VQT uses 36 bins per octave, so neighboring output bins are $1200/36=33.3$ cents apart. A spread of 35 cents is about one bin, which is the narrowest smooth target that the output grid can represent. The upper limit comes from the metric: RPA50 counts a prediction as correct within 50 cents. Therefore, keeping ${\sigma }_{¢}$ below 50 stops the target from applying weight on predictions that the metric treats as errors.
• Residual scale $\gamma =0.66$ (Equation (16)). This is the usual bypass scale of the factored TDNN [37], where a scaled residual keeps a deep stack from relying on the identity path, and it was used here without modification. Forward activation magnitude and gradient flow were recorded for $\gamma \in \{0.0,0.66,1.0\}$. At $0.66$, the layer-wise amplification stays close to one, while $\gamma =1.0$ grows the activation magnitude with depth and $\gamma =0.0$ removes the residual path.
• Latent width $H=18$. The width was reduced until validation accuracy on MDB-stem-synth began to drop. $H=18$ is the smallest width at which there was no performance drop. Section 3.2 varies L from 4 to 10 and shows the effect on accuracy which is still improving at the largest depth, so the model is limited by depth rather than width.
• Kernel size 5. A kernel of size 5 places two taps on each side of the center bin at every dilation, so a block can compare a bin with neighbors above and below it at that scale, whereas $k=3$ gives only one tap per side. In the early blocks, where $d_l=1$, the $\pm 2$ bin span covers $\pm 2/3$ of a semitone at 3 bins per semitone, which is the local pitch neighborhood that the model needs before the wider dilations start linking harmonics. The wider kernel also grows the receptive field faster: with $k=5$ the field covers the full 269-bin range by the eighth block, in contrast with the ninth for $k=3$, which leaves more of the depth budget for refinement rather than for reaching across the band.

2.8.4. Training Configuration

All models are trained with the Adam optimizer [38], at a learning rate of ${10}^{-3}$, halved every 15 epochs, with a mini-batch size of 256, for a total of 100 epochs. Mini-batches come from the octave-balanced sampler of Section 2.7.1, with per-epoch caps of $2\times {10}^5$ voiced and $2\times {10}^5$ unvoiced frames. The consistency terms of Section 2.7.4 use weights of $0.05$, $0.05$ and $0.025$ for the noisy, shifted, and shifted-and-noisy views. They are applied in unidirectional form, with the clean view distribution detached as the target. Training and evaluation use single precision.

3. Results and Discussion

The experiments evaluate pitch estimation under three conditions: clean in-domain validation, controlled architectural comparison between the proposed FrCN and a CNN baseline, and robustness under additive noise. Raw Pitch Accuracy at 50 cents (RPA50) is the fraction of voiced frames whose predicted $f_0$ lies within 50 cents of the reference pitch. Raw Chroma Accuracy at 50 cents (RCA50) applies the same 50-cent tolerance after octave folding, so it measures pitch-class correctness while ignoring octave errors. Mean Error (MnE) and Median Error (MdE) are the mean and median absolute pitch errors, respectively, reported in semitones. Unless otherwise stated, RPA and RCA are reported as percentages, while MnE and MdE are reported in semitones. Higher RPA50 and RCA50 indicate better pitch accuracy and lower MnE and MdE indicate smaller pitch error.

3.1. Clean In-Domain Pitch Estimation

Table 3 summarizes the clean same-dataset results. The proposed FrCN improves over the CNN baseline on all three validation sets. The largest gain is observed on MDB-stem-synth, where FrCN increases RPA50 from 96.1% to 98.8% and reduces mean error from 0.19 to 0.05 semitones. On PTDB, FrCN improves RPA50 by 1.2 percentage points and reduces MnE from 0.40 to 0.33 semitones. MIR-1K is already strong for both models, but FrCN still improves RPA50 and RCA50 while reducing mean error. Figure 6 helps visualize the results from Table 3 as error rate and log-scale semitone error.

Table 3. Clean in-domain pitch-estimation results for CNN and FrCN models. Bold values indicate the best performance for each validation set.

Validation Set	Model	RPA50	RCA50	MnE	MdE
MDB	CNN	96.1	96.4	0.19	0.03
	FrCN	98.8	98.8	0.05	0.01
PTDB	CNN	91.9	93.1	0.40	0.10
	FrCN	93.1	94.1	0.33	0.08
MIR-1K	CNN	96.7	96.8	0.16	0.08
	FrCN	97.5	97.5	0.13	0.08

Figure 6. Clean in-domain error-rate comparison for CNN and FrCN models. The panels show: (A) RPA50 error rate ($1-\mathrm{RPA}50$), (B) RCA50 error rate ($1-\mathrm{RCA}50$), (C) MnE, and (D) MdE for MDB, PTDB, and MIR-1K, with lower values indicating better performance. The MnE and MdE panels use log-scaled y-axes.

The clean in-domain results first show that the gain is not only a noise effect. FrCN improves both threshold-based metrics and continuous error. This means that the model is not only fixing a few large octave mistakes but it is also placing the pitch estimate closer to the reference. This is clear on MDB, where the labels are precise. A likely reason is the larger frequency-axis receptive field of FrCN. Pitch is usually supported by several harmonics, not just one peak. FrCN can use a wider frequency context early in the network, while a local CNN needs more layers before distant harmonics can interact.

These gains are also competitive with recent state-of-the-art pitch trackers while using a much smaller model. Table 4 compares FrCN with published in-domain results across MDB, PTDB, and MIR-1K. The proposed FrCN uses only 17.8k parameters, while the other models range from 130k parameters to tens of millions of parameters. Even with this gap in size, FrCN gives the best or near-best result on the three validation sets. It also gives the lowest reported MnE and MdE wherever those metrics are available. This suggests that the frequency-axis convolution is using the VQT representation well: it gives the model enough context for pitch estimation without needing a large network.

Table 4. Clean in-domain comparison with recent pitch-estimation models. Values for the other models are as reported in their source publications. Missing values indicate metrics not reported in the corresponding source. Bold values indicate the best performance for each validation set.

Validation Set	Model	Parameters	RPA50	RCA50	MnE	MdE
MDB [29]	FrCN (Ours)	17.8k	98.8	98.8	0.05	0.01
	PESTO [11]	130k	97.0	97.1	0.18	0.08
	CREPE [10]	22.2M	97.3	97.4	0.13	0.02
	PESTO-OT [12]	130k	95.3	95.6	–	–
	YOLOPitch [13]	2.8M	99.0	99.1	–	–
PTDB [30]	FrCN (Ours)	17.8k	93.1	94.1	0.33	0.08
	PESTO [11]	130k	89.7	91.2	0.56	0.13
	CREPE [10]	22.2M	87.1	89.9	1.24	0.11
	PESTO-OT [12]	130k	89.0	90.8	–	–
	YOLOPitch [13]	2.8M	90.4	91.4	–	–
MIR-1K [31]	FrCN (Ours)	17.8k	97.5	97.5	0.13	0.08
	PESTO [11]	130k	97.7	98.0	0.21	0.12
	CREPE [10]	22.2M	97.5	98.0	0.23	0.07
	PESTO-OT [12]	130k	97.8	98.1	–	–
	YOLOPitch [13]	2.8M	97.2	97.3	–	–
	DJCM [14]	46.7M	93.72	93.96	–	–
	FCPE [15]	10.6M	96.79	–	–	–
	SLASH [16]	10.1M	96.7	96.7	–	–

The main result from Table 4 is that FrCN gives a better accuracy–size tradeoff than the compared systems. It does not need to cleanly win every metric to be the stronger model for the target setting. On MDB, YOLOPitch is only slightly higher in RPA50 and RCA50, but it uses about 157 times more parameters. On PTDB, FrCN is the best model across all reported metrics. On MIR-1K, FrCN is close to the best RPA50/RCA50 results and gives the lowest reported mean and median error. In other words, the larger models often buy only a small gain in one metric, while FrCN keeps nearly the same or better accuracy with a much smaller model.

It must be noted that since all three corpora are public benchmarks, and the systems compared here report results on them under their own published protocols, the figures are directly usable and are quoted as published rather than re-measured. This method is preferred over re-running the other systems ourselves since a re-implementation configured by us could as easily understate a competing method. The protocols are not identical in every detail, so the accuracy metrics may differ between the cited works, so small differences in the accuracy columns should not be over-interpreted. The parameter counts, on which the efficiency argument of this paper rests, are properties of the published architectures and are unaffected by any of these protocol details.

This is important because pitch estimation is often not used alone. In timbre transfer, voice synthesis, singing voice conversion, and pitch-conditioned audio effects, the $f_0$ estimator is only one part of a larger system. Edge-device pitch-estimation work reports real-time factors around 0.5–0.6 on small hardware, even for models designed to be efficient. If the pitch estimator already takes a large part of the compute budget, the full system becomes harder to run in real time. FrCN is better suited to this use case because it keeps state-of-the-art level accuracy while leaving most of the parameter and compute budget for the rest of the pipeline.

3.2. Effect of Architecture at Matched Parameter Scale

To separate architecture from parameter count, Table 5 compares FrCN and CNN models trained and evaluated on MDB at matched layer counts and parameter budgets. At every parameter size, FrCN gives higher RPA50 and lower mean error than the CNN. The difference is especially clear in the middle of the parameter range: at 12.8k parameters, FrCN obtains 98.5% RPA50, while CNN obtains 93.0%. Even the 11.2k-parameter FrCN outperforms the 17.8k-parameter CNN on all metrics.

Table 5. Clean MDB results for FrCN and CNN models at matched parameter scales. Bold values indicate the best performance for each layer configuration.

Layers	Model	Parameters	RPA50	RCA50	MnE	MdE
4	CNN	7.9k	91.4	95.3	0.76	0.06
	FrCN		93.3	95.2	0.49	0.04
6	CNN	11.2k	92.4	95.8	0.64	0.05
	FrCN		97.4	97.5	0.09	0.02
7	CNN	12.8k	93.0	96.0	0.56	0.04
	FrCN		98.5	98.6	0.06	0.01
8	CNN	14.5k	94.1	95.8	0.42	0.04
	FrCN		98.6	98.6	0.05	0.01
10	CNN	17.8k	96.1	96.4	0.19	0.03
	FrCN		98.8	98.8	0.05	0.01

This result supports the main design choice. FrCN uses dilated frequency-axis convolutions, so its receptive field grows quickly as layers are added. With the dilation schedule used here, the receptive field grows exponentially with depth, while a standard CNN with local filters grows linearly. This difference plays an important role for pitch estimation, since $f_0$ is not dependent only on a local peak but supported by a harmonic pattern across the log-frequency axis. FrCN can compare these regions with fewer layers, while CNN needs more layers to connect them.

Table 2 presents the numbers related to this growth, and also explains why the difference between the two models is largest at medium depth. At $L=4$ both models see less than an octave, $0.97$ against $0.53$, so neither can relate a fundamental to its harmonics and the two perform similarly. By $L=6$, the FrCN covers $3.64$ octaves while the CNN baseline is still at $0.75$, and the difference increases. At $L=7$ the FrCN spans $7.19$ octaves, almost the whole $7.47$ octave band, and the difference is at its largest, which is the row related to the $12.8$k-parameter model in the table. Beyond that the FrCN field already exceeds the band, so added depth brings no new frequency context, while the CNN baseline keeps widening slowly and closes part of the gap. The largest difference therefore appears at the depth where the FrCN has just gained access to the full band and the CNN has not.

Figure 7 also supports this analysis. The CNN error decreases slowly as depth increases, which is similar to its linear receptive-field growth. The FrCN error drops much faster once the dilated stack has enough depth, which is similar to the faster receptive-field growth. This suggests that the main change is not only the parameter count. The model needs enough frequency context to compare distant harmonics and reduce octave errors. This is also seen in the relation between RCA50 and RPA50. CNN often keeps RCA50 higher than RPA50 in small models, which suggests that it finds the pitch class but still makes octave errors. FrCN improves both RPA50 and RCA50 as it grows, so it resolves octave placement better.

Figure 7. Parameter-matched FrCN and CNN comparison on clean MDB validation. Each panel plots error against parameter count, with (A) RPA error rate, (B) RCA error rate, (C) Mean error and (D) Median error. This shows the FrCN advantage is consistent across model size rather than only at the largest configuration. The semitone-error panels use log-scaled y-axes.

3.3. Noise Robustness

Table 6 reports representative in-domain noisy conditions for both architectures. The CNN baseline drops strongly under severe noise. On MDB, CNN RPA50 falls to 28.2% at 0 dB SNR, while FrCN keeps 80.1% RPA50 and 86.2% RCA50. The same pattern appears on PTDB and MIR-1K, where FrCN keeps the RPA50 above 80% at 0 dB SNR. Figure 8 helps visualize the results from Table 6 as log-scale error rate and mean semitone error.

Table 6. In-domain robustness under representative additive-noise conditions. Bold values indicate the best performance for each validation set.

Validation Set	Model	50 dB			20 dB			10 dB			0 dB
		RPA50	RCA50	MnE	RPA50	RCA50	MnE	RPA50	RCA50	MnE	RPA50	RCA50	MnE
MDB	CNN	96.1	96.4	0.19	94.8	95.4	0.31	89.4	90.8	1.11	28.2	34.7	17.48
	FrCN	98.8	98.8	0.05	98.2	98.2	0.08	97.2	97.7	0.15	80.1	86.2	2.27
PTDB	CNN	91.9	93.1	0.40	91.4	92.6	0.42	88.4	89.8	0.57	51.4	55.7	5.86
	FrCN	93.1	94.1	0.33	93.0	94.0	0.34	92.3	93.6	0.38	80.5	84.4	1.47
MIR-1K	CNN	96.7	96.8	0.16	96.6	96.7	0.16	96.3	96.4	0.19	70.7	72.9	3.95
	FrCN	97.5	97.5	0.13	97.5	97.5	0.13	97.3	97.4	0.14	85.2	87.8	1.10

Figure 8. In-domain noise robustness for CNN and FrCN models. The panels report RPA50 error rate, RCA50 error rate, and MnE across clean, 50, 40, 30, 20, 10, and 0 dB SNR conditions for MDB, PTDB, and MIR-1K. All panels use log-scaled y-axes.

The same trend appears in the full clean-to-0 dB SNR sweep. At 17.8k parameters, FrCN averages a 95.7% RPA50 across noise conditions, compared with 85.2% for CNN. At 0 dB SNR, FrCN reaches around 80.0% RPA50, while CNN reaches 28.2%. This shows that the architecture is not only better on clean frames. It also keeps more pitch information when the input is strongly corrupted.

The likely reason is again related to the receptive field. At 0 dB SNR, many local spectral peaks are unreliable because the noise energy is close to the signal energy. A local CNN can be misled by these corrupted peaks because each early layer sees only a small frequency neighborhood. FrCN can use a wider frequency range and compare several harmonics at once. Noise may hide one harmonic, but it is less likely to create a complete harmonic series that is consistent with the same $f_0$. This gives FrCN a better way of rejecting local noise peaks and keeping the pitch estimate stable.

This is ideal for real-time use. In practice, input audio may include accompaniments, room noise, microphone coloration, or other interference. The 0 dB SNR result does not mean that FrCN is unaffected by noise, but it means that the failure mode is less severe. For systems such as transcription, tuning feedback, or pitch-conditioned audio effects, this matters because a moderate error is easier to handle than a collapse into random or octave-wrong predictions.

3.4. Ablation of Noise and Frequency Scaling Augmentation

The augmentation ablation evaluates four training settings: no augmentation, VQT shift-only augmentation, noise-only augmentation, and the proposed joint VQT shift-plus-noise setting. Table 7 and Table 8 report representative noise levels in two ways: MDB-only in-domain performance and average performance over MDB, PTDB, and MIR-1K. Figure 9 visualizes the full error-rate curves.

Table 7. MDB in-domain FrCN augmentation ablation for clean, moderate-noise, and severe-noise validation conditions. Bold values indicate the best performance for each training augmentation.

Training Augmentation	Clean			10 dB			0 dB
	RPA50	RCA50	MnE	RPA50	RCA50	MnE	RPA50	RCA50	MnE
Shift + noise	98.8	98.8	0.05	97.2	97.7	0.15	80.1	86.2	2.27
Noise only	98.3	98.3	0.07	96.1	96.9	0.30	81.3	86.1	2.30
Shift only	98.7	98.7	0.06	87.0	89.7	1.46	64.0	72.5	4.95
No augmentation	98.7	98.8	0.06	81.0	83.8	2.05	57.8	64.5	5.81

Table 8. Overall FrCN augmentation ablation averaged over MDB, PTDB, and MIR-1K for clean, moderate-noise, and severe-noise validation conditions. Bold values indicate the best performance for each training augmentation.

Training Augmentation	Clean			10 dB			0 dB
	RPA50	RCA50	MnE	RPA50	RCA50	MnE	RPA50	RCA50	MnE
Shift + noise	94.3	95.0	0.35	93.5	94.6	0.42	76.7	82.0	2.45
Noise only	93.5	94.3	0.46	92.3	93.4	0.58	79.3	83.4	2.28
Shift only	93.0	94.1	0.58	84.3	87.5	1.67	62.2	70.5	5.08
No augmentation	91.2	92.3	0.62	76.9	80.5	2.35	54.7	61.7	5.75

Figure 9. Effect of VQT shift and noise augmentation on FrCN robustness. The curves report RPA50 error rate, RCA50 error rate, and mean pitch error across clean and noisy validation conditions for MDB, PTDB, and MIR-1K. All panels use log-scaled y-axes.

Using two tables is useful because they answer different questions. Table 7 isolates the in-domain MDB setting, where the training and validation data match. Table 8 then averages over MDB, PTDB, and MIR-1K to show whether the same augmentation choice remains useful beyond one validation set. The MDB table is therefore the controlled in-domain result, while the combined table is the overall robustness result.

The two tables show that the augmentations have different roles. On clean MDB, all models with frequency scaling augmentation or no augmentation are close, but the combined VQT shift-plus-noise model is still the best: it reaches 98.8% RPA50, 98.8% RCA50, and 0.05 semitone MnE in Table 7. In the combined table, the same setting is also best on clean validation, with 94.3% RPA50, 95.0% RCA50, and 0.35 semitone MnE. This suggests that adding noise during training does not damage the clean performance. Instead, when it is paired with frequency scaling augmentation, it acts as a useful regularizer.

At 10 dB SNR, the advantage of using both augmentations becomes clearer. On MDB, VQT shift-plus-noise gives 97.2% RPA50 and 0.15 semitone MnE, while noise-only drops to 96.1% RPA50 and 0.30 semitone MnE. The gap is larger for the weak augmentation settings: VQT shift-only gives 87.0% RPA50 and no augmentation gives 81.0%. The combined table shows the same pattern across datasets, where VQT shift-plus-noise gives 93.5% RPA50 and 0.42 semitone MnE, compared with 92.3% and 0.58 for noise-only. This indicates that moderate noise is not handled only by learning noise invariance. The model also benefits from learning how harmonic patterns move when pitch changes.

At 0 dB SNR, the result changes because the noise is severe enough that noise invariance becomes the dominant requirement. On MDB, noise-only gives the highest RPA50 at 81.3%, but VQT shift-plus-noise is almost tied and gives slightly better RCA50 and MnE, with 86.2% RCA50 and 2.27 semitones MnE. In the combined table, noise-only is best at 0 dB SNR on all three metrics, reaching 79.3% RPA50, 83.4% RCA50, and 2.28 semitones MnE. This does not weaken the combined augmentation result. It shows a tradeoff: noise-only specializes for the hardest noise case, while VQT shift-plus-noise gives the best clean and moderate-noise behavior and remains far above VQT shift-only and no augmentation at 0 dB SNR.

Figure 9 supports this interpretation across the full SNR sweep. The curves show that VQT shift-plus-noise is most consistent through the clean and intermediate noise region, including 50 dB SNR, 20 dB SNR, and 10 dB SNR. Noise-only becomes most competitive only at the lowest SNR. A useful hypothesis is that frequency scaling augmentation improves pitch equivariance: when the log-frequency pattern shifts, the prediction should shift by the same interval. This also helps cross-dataset transfer. Compared with no augmentation, VQT shift-only improves clean out-of-domain RPA50 from 84.7% to 87.5% on PTDB and from 90.1% to 92.8% on MIR-1K. When VQT shift is combined with noise, the gain is stronger on MIR-1K: clean RPA50 improves from 93.6% with noise-only to 96.3%, and 10 dB SNR RPA50 improves from 93.2% to 96.2%. PTDB is more mixed in RPA50, but the combined model has lower MnE than noise-only in the clean and 10 dB SNR cases. Noise augmentation improves invariance: when unrelated spectral energy is added, the prediction should stay fixed. The combined model learns both properties, so it gives the best general-purpose robustness. At 0 dB SNR, however, the invariance objective is so dominant that the noise-only model can slightly overtake it in the averaged result.

3.5. Out-of-Domain Generalization

The most useful cross-dataset setting is training on MDB and testing on other datasets. MDB has accurate synthesized stem annotations and includes both vocal and instrumental sources. Table 9 compares MDB-trained FrCN with PESTO on clean out-of-domain data. On MIR-1K, FrCN improves over PESTO on all four metrics, increasing RPA50 from 94.6% to 96.3% and reducing MnE from 0.57 to 0.35 semitones. On PTDB, PESTO is slightly higher in RPA50 and RCA50, but FrCN remains close while using about one seventh of the parameters. Figure 10 helps visualize the results from Table 9 as error rate and mean semitone error.

Table 9. Clean out-of-domain comparison when models are trained on MDB and validated on PTDB or MIR-1K. Additional published models can be inserted as extra rows within each validation set. Bold values indicate the best performance for each validation set.

Validation Set	Model	Parameters	RPA50	RCA50	MnE	MdE
PTDB	FrCN	17.8k	87.7	89.3	0.65	0.15
	PESTO	130k	88.3	89.9	0.62	0.13
MIR-1K	FrCN	17.8k	96.3	97.0	0.35	0.10
	PESTO	130k	94.6	96.1	0.57	0.14

Figure 10. Clean MDB-trained out-of-domain comparison between FrCN and PESTO. The panels show: (A) RPA50 error rate, (B) RCA50 error rate, (C) Mean error, and (D) Median error for PTDB and MIR-1K, emphasizing where MDB-trained FrCN generalizes better to musical data and where speech-domain transfer remains competitive. The Mean and Median error panels use log-scaled y-axes.

Table 10 reports the same MDB-trained FrCN under additive noise. The model transfers well to MIR-1K: clean RPA50 is 96.3%, and it persists at 96.2% for 10 dB SNR. PTDB is harder because it contains speech rather than singing or instrumental stems, but FrCN is still stable through moderate noise, with 87.2% RPA50 at 10 dB SNR. Figure 11 helps visualize the results from Table 10 as log-scale error rate and mean semitone error.

Table 10. Out-of-domain FrCN noise robustness when trained on MDB and validated on PTDB or MIR-1K. Bold values indicate the best performance for each validation set.

Validation Set	Condition	RPA50	RCA50	MnE	MdE
PTDB	Clean	87.7	89.3	0.65	0.15
	20 dB	87.6	89.3	0.66	0.16
	10 dB	87.2	89.0	0.69	0.16
	0 dB	71.7	76.2	2.44	0.30
MIR-1K	Clean	96.3	97.0	0.35	0.10
	20 dB	96.5	97.1	0.35	0.10
	10 dB	96.2	97.1	0.43	0.10
	0 dB	71.3	78.8	4.51	0.27

Figure 11. MDB-trained FrCN out-of-domain noise robustness. The panels report: (A) RPA50 error rate, (B) RCA50 error rate, (C) Mean error, and (D) Median error across clean, 50, 40, 30, 20, 10, and 0 dB SNR conditions for PTDB and MIR-1K. All panels use log-scaled y-axes.

The MDB-to-MIR result suggests that FrCN is not just memorizing one dataset. MDB and MIR-1K are different, but both contain musical pitch trajectories and harmonic structure. The transfer result suggests that the filters learn reusable log-frequency patterns. MDB-to-PTDB is harder because speech has different phonation, formants, and voicing behavior. The smaller gap between FrCN and PESTO on PTDB is therefore likely due to domain mismatch, not only architecture.

The out-of-domain noise curves add another point. The model transfers not only in clean audio but also under moderate noise. Down to 10 dB SNR, the MDB-trained FrCN is stable on both PTDB and MIR-1K. This means that the augmentation strategy is not just helping in-domain noise robustness. The larger drop at 0 dB SNR is expected because the model is both outside its training domain and under strong interference. Even then, the model maintains useful performance, which is important when labels for the target domain are limited.

3.6. Voicing Results

The voicing head was evaluated using recall, false alarm rate (FAR), error rate (ER), accuracy, and F1 score. FAR is computed as the fraction of unvoiced frames that are predicted as voiced, i.e., $\mathrm{FP}/(\mathrm{FP}+\mathrm{TN})$. Table 11 reports the strongest FrCN voicing setting, where the model is trained using all three datasets and evaluated on each validation set under clean, 10 dB SNR, and 0 dB SNR conditions.

Table 11. FrCN voicing results when trained on all datasets and validated on each dataset. FAR denotes false alarm rate and ER denotes error rate. Bold values indicate the best performance for each validation set.

Validation Set	Condition	Recall	FAR	ER	Accuracy	F1
MDB	Clean	99.8	0.3	0.2	99.8	99.7
	10 dB	95.1	6.0	5.5	94.5	94.0
	0 dB	66.2	6.4	18.7	81.3	76.1
PTDB	Clean	94.6	2.8	3.4	96.6	92.8
	10 dB	89.0	1.8	4.0	96.0	91.3
	0 dB	60.5	1.7	10.5	89.5	72.9
MIR-1K	Clean	98.3	7.5	3.5	96.5	97.5
	10 dB	97.4	5.1	3.4	96.6	97.6
	0 dB	87.3	4.8	10.4	89.6	92.2

The in-domain and multi-domain voicing results are strong in clean and moderate-noise conditions. On clean MDB, the model reaches 99.8% recall, 0.3% FAR, 0.2% ER, 99.8% accuracy, and 99.7% F1. On PTDB and MIR-1K, the F1 score for the clean data remains high at 92.8% and 97.5%, respectively. At 10 dB SNR, the model remains stable: the F1 score is 94.0% on MDB, 91.3% on PTDB, and 97.6% on MIR-1K. This suggests that the voicing head can distinguish voiced and unvoiced frames well when the training set covers the target domain and the noise is not extreme.

At 0 dB SNR, the main change is a drop in recall rather than a large increase in FAR. MDB recall falls from 99.8% in clean audio to 66.2%, while FAR remains close to 6%. PTDB shows a similar pattern, with recall falling to 60.5% and FAR staying below 2%. MIR-1K is more robust, keeping 87.3% recall and 92.2% F1 at 0 dB SNR. A likely reason is that singing voice frames in MIR-1K often contain stronger and more sustained harmonic structure than speech frames, so the voiced/unvoiced boundary remains easier to detect under severe noise. In contrast, speech contains shorter voiced regions, weaker onsets, and more low-energy transitions, which are easier to suppress when the input is strongly corrupted.

Across validation sets, ER remains below 20% even at 0 dB SNR for the all-trained FrCN. This is important because the voicing task is evaluated under the same severe additive-noise conditions as pitch estimation. The model does not simply mark every frame as voiced: FAR stays low on MDB and PTDB, and remains below 8% on MIR-1K. Instead, the main failure mode at 0 dB SNR is missed voiced frames, especially on MDB and PTDB. This suggests that the voicing head becomes cautious when harmonic evidence is weak or masked by noise.

These results show that the voicing head is accurate when the training data covers the validation domains, but its decision threshold becomes conservative under severe noise. When it predicts a frame as voiced, it is usually correct, but at 0 dB SNR it misses more voiced frames. This may happen because the model is mainly optimized for accurate $f_0$ estimation on voiced frames. It may also learn that false positives are costly in low-energy or noisy regions, since predicting voiced frames without reliable pitch evidence can lead to unstable $f_0$ estimates. For applications where false voiced detections are worse than missed voiced frames, this behavior can be useful. For transcription or vocal analysis, higher recall would be better.

3.7. Voicing Operating Points

Table 11 was generated by applying a single threshold of $0.5$ to the voicing probability, which leaves the open question of whether the drop in the recall, at 0 dB SNR, is a property of that threshold or that of the representation learned by the model. To separate the two, the voicing probability was recorded for every frame of every validation set and the threshold was then swept over the recorded values, so that no retraining is involved and that the underlying model is the same one which was reported in Table 11.

Two main points may be inferred from Table 12. The first is that the information needed to separate voiced from unvoiced frames is still present at 0 dB SNR. The threshold-free ROC area under the curve stays at $0.938$ on MDB, $0.971$ on PTDB, and $0.978$ on MIR-1K, versus $1.000$, $0.986$, and $0.995$ on clean audio. It does not collapse toward $0.5$, as it would have if the distinction had been lost.

Table 12. Voicing decision at three operating points. $\tau$ is the threshold applied to the voicing probability. $\tau =0.5$ is the value used in Table 11 and that column reproduces it to within $0.1$ points, the additive noise being redrawn in this evaluation. Recall and the false alarm rate (FAR) are reported in percentages. The ROC area under the curve is threshold-free and measures how well the voicing probability separates voiced frames from unvoiced frames.

Validation Set	Condition	ROC AUC	$\mathit{\tau }=0.1$		$\mathit{\tau }=0.5$		$\mathit{\tau }=0.7$
			Recall	FAR	Recall	FAR	Recall	FAR
MDB	Clean	1.0000	100.0	0.5	99.8	0.3	99.6	0.2
	10 dB SNR	0.9896	99.1	18.1	95.1	6.0	91.4	3.1
	0 dB SNR	0.9377	91.4	19.6	66.1	6.4	53.5	3.2
PTDB	Clean	0.9859	97.9	6.9	94.6	2.8	91.2	1.9
	10 dB SNR	0.9845	97.0	5.4	89.0	1.8	81.4	1.1
	0 dB SNR	0.9705	91.4	6.0	60.5	1.7	43.9	1.0
MIR-1K	Clean	0.9949	99.8	21.5	98.3	7.5	96.9	4.4
	10 dB SNR	0.9949	99.7	17.9	97.4	5.2	95.2	2.6
	0 dB SNR	0.9778	98.9	19.3	87.3	4.9	75.9	2.4

The second takeaway is that the recall in Table 11 is recoverable by moving the threshold, at a cost that differs among corpora. Lowering $\tau$ from $0.5$ to $0.1$, at 0 dB SNR, raises recall from $66.1\%$ to $91.4\%$ on MDB, from $60.5\%$ to $91.4\%$ on PTDB, and from $87.3\%$ to $98.9\%$ on MIR-1K, respectively. On PTDB, this costs very little, since the false alarm rate rises only from $1.7\%$ to $6.0\%$. On MDB and MIR-1K the same move raises the false alarm rate to about $20\%$. Maximizing F1 at 0 dB SNR, instead of fixing $\tau =0.5$, leads to an improvement from $72.9\%$ to $86.7\%$ on PTDB, from $76.0\%$ to $84.8\%$ on MDB, and from $92.2\%$ to $95.8\%$ on MIR-1K, at thresholds of $0.10$, $0.09$, and $0.18$, respectively.

The behavior described in Section 3.6 is, therefore, mainly a property of a fixed and conservative decision boundary rather than an intrinsic limit of the learned representation under severe noise. This matters for the noted application. A transcription or vocal analysis system, which prefers recall over precision, should not use $0.5$ under severe noise. A threshold near $0.1$ recovers most of the lost recall, and on speech it does so at a false alarm rate still below $7\%$. The threshold is a deployment choice that can be set per application, and the clean-condition results are almost unaffected by it, since on clean audio recall changes by at most one point across the whole range from $0.1$ to $0.7$.

3.8. Statistical Reliability of the Reported Margins

The above results come from single training runs, so some margins are small. This section checks whether the FrCN advantage over the CNN baseline exceeds the variation the evaluation can produce. Both models are run on the same files, so the comparison is paired and may be tested directly on the per-file differences. For every file we take the RPA50 of both models and apply the one-sided Wilcoxon signed-rank test [39], which asks whether FrCN is higher without assuming normally distributed differences. Two further quantities are reported. First is the matched-pairs rank-biserial correlation r [40], which is $+1$ when FrCN is better on every file, 0 when the wins are split evenly, and $-1$ when the baseline always wins. It describes how consistent the difference is, which a p value on several thousand files does not. The second value is a $95\%$ confidence interval on the difference in the pooled RPA50, obtained by resampling files with replacement, i.e., the bootstrap [41], applied here at the level of whole files as is standard when scoring speech and audio systems [42]. Within each set of tests, the p values are corrected for multiple comparisons using the Holm method [43]. The models are the same checkpoints as in the preceding sections, re-evaluated for per-file scores.

Table 13 reports the in-domain comparison. Every difference favors FrCN, with every p value far below any conventional threshold. The smallest margin in the paper, the $1.2$ point gain on PTDB, is measured over 4718 recordings and has a confidence interval of $(1.16,1.31)$, so it is well clear of zero. Its rank-biserial correlation of $0.57$ is the lowest in the table: on PTDB the improvement is real but not obtained on every recording, whereas on MDB-stem-synth r is $1.000$ and FrCN is ahead almost everywhere. Under noise the differences become much larger, between 15 and 53 points.

Table 13. Paired comparison of FrCN against the CNN baseline on in-domain validation data. Each test pairs the two models on the same files and applies the one-sided Wilcoxon signed-rank test. $\Delta$ is the difference in pooled RPA50 in percentage points, with a 95% bootstrap interval over files. r is the matched-pairs rank-biserial correlation. p values are Holm-corrected within each condition.

Condition	Validation Set	RPA50 (%)		$\mathbf{\Delta }$ (95% CI)	r	p
		CNN	FrCN
Clean	MDB	96.2	98.6	2.45 (2.08, 2.87)	1.000	<${10}^{-10}$
	PTDB	92.1	93.3	1.24 (1.16, 1.31)	0.569	<${10}^{-10}$
	MIR-1K	96.7	97.5	0.83 (0.76, 0.90)	0.855	<${10}^{-10}$
0 dB SNR	MDB	27.9	80.8	52.87 (48.52, 57.29)	0.999	<${10}^{-10}$
	PTDB	51.4	80.3	28.83 (28.55, 29.10)	1.000	<${10}^{-10}$
	MIR-1K	70.0	85.0	15.03 (14.35, 15.71)	0.972	<${10}^{-10}$

Table 14 repeats the test at each matched parameter scale, on the two datasets for which a depth sweep was trained. The advantage is present and significant at every depth on both, not only at the largest model. On MDB-stem-synth the clean gain runs from $1.91$ to $5.28$ points and the 0 dB gain from $2.72$ to $63.68$ points; on MIR-1K the corresponding ranges are $0.19$ to $1.06$ and $3.16$ to $20.35$ points. The rank-biserial correlation stays above $0.86$ everywhere except the smallest MIR-1K model, where at $L=4$ the two architectures are closest and r falls to $0.500$, meaning that at that scale the advantage is real but is obtained on a bare majority of recordings rather than on almost all of them. PTDB-TUG is not tabulated here because models were trained on it only at $L=10$; that comparison appears in Table 13, over 4718 recordings.

Table 14. Paired comparison of FrCN against the CNN baseline at matched parameter scales, on the two datasets for which a full depth sweep was trained. Columns are as in Table 13. The Holm-corrected p value is not tabulated because every comparison in this table gives $p<{10}^{-10}$. PTDB-TUG is not included because models were trained on it only at $L=10$ and that comparison is given in Table 13.

Condition	Validation Set	Layers	RPA50 (%)		$\mathbf{\Delta }$ (95% CI)	r
			CNN	FrCN
Clean	MDB	4	91.4	93.3	1.91 (1.56, 2.29)	0.961
		6	92.5	97.4	4.92 (4.12, 5.86)	0.999
		7	93.1	98.3	5.28 (4.51, 6.16)	1.000
		8	94.2	98.5	4.30 (3.73, 4.94)	1.000
		10	96.2	98.6	2.45 (2.08, 2.87)	1.000
	MIR-1K	4	96.2	96.4	0.19 (0.16, 0.22)	0.500
		6	96.4	97.4	1.02 (0.93, 1.11)	0.905
		7	96.4	97.5	1.06 (0.97, 1.15)	0.911
		8	96.5	97.5	1.01 (0.92, 1.09)	0.902
		10	96.7	97.5	0.83 (0.76, 0.90)	0.855
0 dB SNR	MDB	4	11.2	13.9	2.72 (2.18, 3.28)	0.882
		6	12.9	74.4	61.44 (57.94, 64.98)	1.000
		7	13.0	76.7	63.68 (59.85, 67.56)	1.000
		8	19.9	78.8	58.96 (54.41, 63.58)	0.999
		10	27.9	80.8	52.87 (48.52, 57.29)	0.999
	MIR-1K	4	62.3	65.5	3.16 (2.98, 3.35)	0.865
		6	64.2	84.5	20.35 (19.53, 21.19)	0.984
		7	65.2	82.3	17.14 (16.34, 17.94)	0.959
		8	67.0	85.2	18.14 (17.44, 18.84)	0.987
		10	70.0	85.0	15.03 (14.35, 15.71)	0.972

These tests describe how much of the difference can be attributed to the particular files used for evaluation. They do not describe how much the result would move if a model were trained again. To estimate that, two additional models were trained on MIR-1K with the $L=10$ configuration, one for each architecture, differing from the originals only in the random seed used for initialization, data ordering, and the train/validation split. Between the two runs, RPA50 moved by $0.03$ points for FrCN and $0.01$ points for the CNN baseline on clean audio. This is a two-run estimate and should be read as an indication of scale rather than a variance estimate, but it does show that the differences reported in this paper are not an artifact of a single training run.

3.9. Computational Cost and Edge Deployment

In this section we report the computational cost and the measured latency of the full pipeline on two embedded platforms, in order to assess the viability of the proposed system on edge devices. A low parameter count alone does not give low latency, since memory access is also important. Wide dilations, in particular, read positions that are far apart, which is the least cache-friendly pattern in the network. Therefore, both cost and measured time are provided.

3.9.1. Theoretical Complexity

As shown in Section 2.8.2, the $17.8$k parameter model requires $4.60$ MMAC, or $9.20$ MFLOP, per frame at the 269-bin VQT resolution. At the 10 ms hop, this is $0.46$ GMAC/s, or $0.92$ GFLOP/s, to run in real time. Since dilation only changes which positions a kernel reads, the CNN baseline has the same cost, so the accuracies reported in Section 3.1, Section 3.2, Section 3.3, Section 3.4 and Section 3.5 are produced at the same cost.

The memory footprint is correspondingly small. The $17,787$ trainable parameters and the 370 batch-normalization statistics occupy 71 KiB in single precision, and 36 KiB at half precision. The largest intermediate tensor is one $H\times B$ feature map, which is 19 KiB per frame in streaming mode and $4.7$ MiB for a block of 256 frames. Therefore, during the streaming operation, the parameters and the working set, in whole, remain in the hundreds of kilobytes which is small enough to stay resident in the last-level cache of both devices. This removes external memory bandwidth as a constraint. For reference, the peak device memory measured for the same pipeline on a desktop GPU over 30 s of audio was 44 MiB, most of which is the framework overhead rather than model state.

3.9.2. Measurement Protocol

The full pipeline is timed, covering the complex VQT front-end, magnitude extraction, the model forward pass, and the pitch and voicing decoding. Measurements are taken on a Raspberry Pi 5 and a NVIDIA Jetson Orin NX, the latter an 8-core Arm Cortex-A78AE at 2 GHz, both on the CPU in single precision. No quantization, operator fusion, or specialized inference runtime is used. Each setting is run with 5 warm-up iterations and timed over 20 repetitions with a wall-clock timer, and the medians are reported. The real-time factor (RTF) is the processing time divided by the duration of the audio, so $\mathrm{RTF}<1$ means real-time operation and $1/\mathrm{RTF}$ is the margin over real time.

The analysis is performed in two modes. In streaming mode, one frame is processed at a time, which keeps the latency to a single hop but pays the per-call overhead on every frame. In buffered mode, frames are processed in blocks of 256, i.e., $2.56$ s of audio, as in file-based use.

3.9.3. Measured Latency

Table 15 provides the results for the $L=10$ model. Both platforms run in real time, in both modes with room to spare. In the hardest setting, namely streaming one frame at a time, Raspberry Pi 5 uses $30\%$ of the 10 ms frame budget and leaves the rest for the other processes. In buffered mode, the two devices run at a speed, $5.8$ and $21.4$ times real time respectively.

Table 15. Measured latency of the complete pipeline for the $17.8$k-parameter FrCN ($L=10$), single precision on CPU. Streaming figures are the model forward pass, measured per frame. Buffered figures are the complete pipeline over 30 s of audio in blocks of 256 frames. Medians were over 20 repetitions.

Platform	Streaming (1 Frame)			Buffered (256 Frames)
	ms/Frame	RTF	× Real Time	RTF	× Real Time	Frames/s
Raspberry Pi 5 (CPU)	3.01	0.301	3.3	0.173	5.8	579
Jetson Orin NX (CPU)	2.78	0.278	3.6	0.047	21.4	2141

Table 16 splits the buffered measurements by stage. The VQT front-end takes about $5\%$ of the end-to-end time on both platforms and the decoding step well under $1\%$, so most of the time is spent in the network rather than in feature extraction or post-processing. The wide dilations also add no large memory-access penalty since the measured forward time follows the operation count.

Table 16. Stage decomposition of the buffered end-to-end measurement ($L=10$, 30 s of audio, blocks of 256 frames). Medians in milliseconds.

Platform	VQT Front-End	Model Forward	Decoding	End-to-End
Raspberry Pi 5 (CPU)	262.8	4925.2	3.1	5183.9
Jetson Orin NX (CPU)	79.3	1286.5	1.5	1401.4

Jeong et al. [24], cited in Section 1, address the same problem of pitch estimation on edge devices and evaluate on a Jetson AGX Orin and a Jetson Orin NX, reporting CPU real-time factors of $0.5557$ and $0.5850$ for their own $5.6$ M-parameter network. Their definition of the real-time factor matches the one used here: processing time divided by the duration of the audio at the same 10 ms frame interval. Also, their edge measurements use a block of 128 frames for every method on which they report. Therefore, the values in their comparisons are mutually consistent and correspond to the buffered mode of Table 15. A pitch estimator alone therefore consumes about $57\%$ of the real-time budget, which leaves little room for the synthesis, effects, or transcription stages it feeds, and it is the reason that the introduction identifies the per-frame cost rather than the accuracy as the limiting factor for deployment.

Jeong et al. [24] provide timing analysis and results for different techniques, run on a Jetson Orin NX. Although they also provide results for the GPU version which is not relevant to our research since the main point of our research is to build architectures for edge devices without GPU. Table 17 places the proposed model in that context using the numbers reported in [24]. Comparisons are made on the same device, namely a Jetson Orin NX, and at the same 10 ms frame interval. On that device, the proposed model needs $0.047$ of the real-time budget, which is about 12 times lower than the $0.585$ of the most efficient network in their comparison and roughly 60 times lower than CREPE, which does not reach real time on the CPU at all. The real-time factor of the proposed model is also slightly below the classical DSP tracker pYIN, at $0.047$ versus $0.062$. The block sizes differ slightly since their figures use 128 frames throughout while the $0.047$ quoted here is the complete pipeline measured over blocks of 256 frames. The difference is immaterial at this scale: a smaller block amortizes the per-call overhead over fewer frames, so carrying the forward-pass increment of Table 18 onto the end-to-end measurement places a block of 128 at about $0.050$ rather than $0.047$, which leaves the comparison unchanged. Therefore, a neural estimator with a learned front-end is no more expensive than a non-parametric signal-processing baseline and at $17.8$k parameters, about 315 times fewer than the best parametric model.

Table 17. CPU real-time factor on Jetson Orin NX hardware. Values for the compared methods are as reported by Jeong et al. [24] for a Jetson Orin NX CPU, measured with a block of 128 frames for every method so that their values are mutually consistent. The FrCN row is the buffered measurement of Table 15 on the same device. Lower is faster, and $\mathrm{RTF}<1$ is real time. Bold values indicate the best performance as well as lowest parameter neural network model.

Method	Type	Parameters	RTF (CPU)
CREPE [10]	Neural	22.2 M	2.815
Jeong et al. [24]	Neural	5.6 M	0.585
pYIN [8]	DSP	—	0.062
10-layer FrCN (ours)	Neural	17.8 k	0.047

Table 18. Effect of block size at $L=10$, single precision on CPU. Algorithmic latency is the time that must elapse before a block is complete, i.e., the block size times the 10 ms hop. This table times the model forward pass alone, whereas Table 15, Table 16 and Table 19 time the complete pipeline, so the real-time factors here are correspondingly lower. A larger block amortizes the per-call overhead over more frames, so the cost per frame falls as the block grows. Medians are over 20 repeats and at a block of 256 the slowest of the 20 lies within $1.2\%$ of the median on the Raspberry Pi 5 and $8.0\%$ on the Jetson Orin NX.

Block (Frames)	Algorithmic Latency (ms)	Raspberry Pi 5 (CPU)			Jetson Orin NX (CPU)
		ms/Frame	RTF	× Real Time	ms/Frame	RTF	× Real Time
1	10	3.01	0.301	3.3	2.78	0.278	3.6
8	80	1.81	0.181	5.5	1.13	0.113	8.9
32	320	1.69	0.169	5.9	0.60	0.060	16.7
128	1280	1.70	0.170	5.9	0.46	0.046	21.6
256	2560	1.69	0.169	5.9	0.43	0.043	23.4
1024	10,240	1.64	0.164	6.1	0.29	0.029	35.0

Even in the strictest frame-by-frame streaming mode, where no cost is amortized over a block, the proposed model needs $0.278$ of the budget on the Jetson Orin NX and $0.301$ on the Raspberry Pi 5, which is still about half of the $0.585$ reported for buffered inference on comparable hardware. Two qualifications should be attached. The real-time factors of Jeong et al. include data loading, which the measurements here exclude, so a small part of the difference is accounting rather than computation. And the figures here were obtained without quantization, operator fusion, or conversion to a dedicated inference runtime, all of which are normally applied before deployment and would only widen the margin. The conclusion the comparison supports is that the model leaves most of the compute budget to the rest of the system, which is the property the introduction argued was missing.

3.9.4. Deployment Choices: Streaming, Block Size and Depth

The real-time factor alone does not capture the difference between the two modes. Buffered operation cannot emit a result until the block has been filled, so a block of 256 frames carries an inherent wait of $2.56$ s, regardless of the speed of the model. This wait is irrelevant for a file which is processed offline. But for interactive use, such as tuning feedback, live effects, or a vocal monitor, it is the quantity that the user perceives. Streaming reduces the wait to a single 10 ms hop, at the cost of paying the per-call overhead on every frame: on the Raspberry Pi 5 the time per frame rises from $1.69$ ms in blocks of 256 to $3.01$ ms one frame at a time. That cost is affordable, since at $L=10$ the model uses $30\%$ of the frame budget on the Raspberry Pi 5 and $28\%$ on the Jetson Orin NX. The reported numbers in Table 15 are for the model forward pass and scaling the front-end and decoding costs. Table 16 presents the same under one frame setting and add about $0.09$ ms and $0.03$ ms, respectively. For streaming the worst case matters more than the average, because a frame that overruns the hop is a dropped frame, and the dispersion is small: at $L=10$ the 95th percentile is $3.08$ ms against a median of $3.01$ ms on the Raspberry Pi 5, and $3.91$ ms against $2.78$ ms on the Jetson Orin NX, so both stay inside the budget.

Table 18 and Table 19 give the two deployment choices, block size and depth. The two are timed differently and should be read accordingly: Table 18 times the model forward pass alone, whereas we have Table 19, like Table 15 and Table 16, times the complete pipeline. At $L=10$ with a block of 256 frames this is the difference between $0.169$ and $0.173$ on the Raspberry Pi 5 and between $0.043$ and $0.047$ on the Jetson Orin NX, the gap in each case being the front end and the decoding step itemized in Table 16, which the forward-pass timing excludes. Block size behaves differently on the two devices. On the Raspberry Pi 5 the per-frame cost falls from $3.01$ ms to $1.69$ ms as the block grows to 32 frames and then stops improving, so larger blocks only add waiting time. On the Jetson Orin NX the cost keeps falling out to 1024 frames, from $2.78$ ms to $0.29$ ms, so there the trade is real. An interactive system should therefore use a block of one to eight frames on either device, and a throughput-bound system has nothing to gain on the Raspberry Pi 5 beyond 32.

Table 19. End-to-end real-time factor as a function of depth (30 s of audio, blocks of 256 frames, CPU, single precision, FrCN). Lower is faster. $1/\mathrm{RTF}$ is the achieved speed relative to real time.

Layers L	Parameters	Raspberry Pi 5 (CPU)		Jetson Orin NX (CPU)
		RTF	× Real Time	RTF	× Real Time
4	7851	0.074	13.6	0.022	46.5
5	9507	0.092	10.9	0.026	38.7
6	11,163	0.109	9.2	0.030	33.5
7	12,819	0.125	8.0	0.034	29.2
8	14,475	0.141	7.1	0.039	26.0
9	16,131	0.157	6.4	0.043	23.2
10	17,787	0.173	5.8	0.047	21.4

Depth scales close to linearly on both platforms, as expected from the constant cost per block, and every configuration fits the budget in both modes. On the Raspberry Pi 5 the buffered real-time factor goes from $0.074$ at $L=4$ to $0.173$ at $L=10$, and frame-by-frame the time per frame goes from $1.62$ ms to $3.01$ ms, that is from $6.2$ to $3.3$ times real time. A system that needs more headroom can therefore trade depth for about a factor of $2.3$ in processing time, at the accuracy cost given in Table 5. Taken together, the measurements show that the model meets the real-time constraint on both platforms with a clear margin, in the strictest frame-by-frame mode and without any deployment optimization, and leaves most of the compute budget for the rest of the system.

4. Conclusions

This work introduced a low-parameter Frequency Convolution Network for monophonic $f_0$ and voicing estimation from VQT features. The main design choice was to apply dilated convolution along the log-frequency axis, so that the model can use harmonic information over a wide frequency range without a large parameter count. The results show the importance of this receptive-field structure, i.e., FrCN improves over a parameter-matched CNN baseline on clean in-domain data and under additive noise. The 17.8k parameter FrCN also remains competitive with recent pitch-estimation models that use many more parameters, which is important when pitch estimation is only one part of a larger real-time audio system. On a Raspberry Pi 5 and a Jetson Orin NX CPU, the complete pipeline runs at real-time factors of $0.173$ and $0.047$, and at $0.301$ and $0.278$, when one frame is processed at a time.

The augmentation study showed that noise and frequency scaling augmentations play different roles. Noise augmentation improves robustness at the hardest SNR levels, while frequency scaling augmentation helps the model learn pitch-equivariant behavior on the log-frequency representation. Their combination gives the best clean and moderate-noise performance and remains strong at 0 dB SNR. The model also transfers well from MDB to MIR-1K and remains competitive on PTDB, although speech-domain transfer is harder. The voicing head performs well when the training data covers the validation domains, but it becomes more conservative under severe noise. The threshold analysis of Section 3.7 shows that most of the recall lost at 0 dB SNR is recoverable by moving the decision boundary. Therefore, the voicing operating point is best chosen per application rather than keeping it fixed. Class-balanced voicing losses will be explored in the future for more robustness to threshold variation. Additionally, a C++ implementation with quantization, operator fusion and device-level CPU feature compiler optimization will be implemented.