Spectral Ripples in Normal and Electric Hearing Models

Abstract

Devising a psychophysical test to assess spectral resolution has not been easy. Two tests that have been used previously are the spectral ripple test and the spectral-temporally modulated ripple test (SMRT). Over time, questions have been raised about the validity of these tests. We introduce a new computational electric hearing (with a cochlear implant, CI) model that can simulate how sound is transferred through a speech processor and is received by the cochlear nerve fibers. With this electric hearing model and a normal hearing model, we investigated whether the known limitations of these tests can be detected. For the spectral ripple test, we could show the limitations in the output of the CI, the information conveyed to the cochlear nerve, to estimate the threshold, and show the benefit of current steering. In addition, we reproduced the aliasing effect with normal hearing in the SMRT, as well as the reduced ripple resolution in CI users. Our computational modeling framework can serve as a first-step assessment of the validity of new psychophysical tests. Moreover, it could be used to test new speech coding strategies.

1. Introduction

Cochlear implants (CIs) are advanced neurostimulators that significantly improve speech perception for the deaf across all age groups [1,2]. Despite these benefits, CI users often struggle with understanding speech in noisy environments [3,4]. In addition, though CI users appreciate music, the pitch and timbre are not conveyed properly [5]. These challenges arise from the complexity of auditory perception, which involves multiple interrelated qualities.

CIs utilize the tonotopic organization of the cochlea, in which high frequencies are processed at the base and low frequencies at the apex. However, the insertion depth of the electrode array is limited, resulting in lower frequencies being presented at suboptimal locations, typically within two cochlear turns of the round window. After implantation, the auditory system is assumed to adapt its tonotopic organization to align with the frequency allocation of the CI [6,7].

Psychophysical tests have been developed to assess the auditory capabilities of CI users, such as loudness and pitch discrimination or forward masking effects. However, the process of designing these tests is complex, and it is crucial to identify and address confounding factors, preferably before involving participants. Cochlear modeling is a valuable tool for this purpose, allowing preliminary evaluations of psychophysical tests before conducting clinical trials.

Advances in computational power have enabled the creation of highly detailed models that simulate auditory responses at the fiber level for the entire cochlea. These models provide insights into what listeners perceive and can reveal potential limitations or unintended cues in psychophysical tests. In this paper, we evaluate two psychophysical tests found to contain unintentional cues during clinical evaluations. We propose a modeling-based approach to enhance the reliability of these tests before their clinical use, reducing the need for patient involvement and improving resource efficiency.

1.1. Spectral Ripple Test

Assessing spectral resolution through psychophysical testing has proven to be challenging. Early approaches often utilized spectral ripple paradigms, which involve modulating a signal’s spectral content [8,9]. Won et al. [10], developed the spectral ripple discrimination test, in which rippled stimuli with sinusoidal envelopes on a logarithmic frequency scale (100 Hz to 5000 Hz) are generated at varying ripple densities, measured in ripples per octave (RPO). In this test, participants use a three-alternative forced-choice (3AFC) format to distinguish between a standard reference stimulus and an inverted (phase-reversed) stimulus. However, its validity for CI users has been questioned. Studies suggest that ripple peaks align with CI filter spacing as the ripple density increases, which confounds the results because the filter spacing does not adequately sample the spectral peaks and valleys at high ripple densities [11,12,13]. Anderson et al. [11] analyzed this limitation using the Nyquist–Shannon sampling theorem, which states that the filter spacing must be less than half the ripple rate to reconstruct the signal accurately. Similarly, Gifford et al. [13] noted that performance above 2 RPO is constrained by the frequency assignment of CIs and hypothesized that CI users rely on both spectral and temporal processing. Anderson et al. [12] also observed non-monotonic behavior in spectral modulation transfer functions, suggesting that some participants unintentionally shift between perceptual cues. Winn and O’Brien [14] proposed that CI limitations and ripple distortions above 2 RPO exacerbate these issues. Interestingly, Drennan et al. [15] demonstrated that the current steering speech coding strategy (Fidelity 120 by Advanced Bionics) improves performance, achieving 3.42 RPO compared to 2.31 RPO with the HiResolution (HiRes) strategy.

1.2. Spectral-Temporally Modulated Ripple Test (SMRT)

To address the challenges noted above, Aronoff and Landsberger [16] introduced the spectral-temporally modulated ripple test (SMRT), incorporating temporal modulation to minimize confounding factors, such as loudness cues, spectral centroids, and changes to spectral edges [17]. SMRT stimuli are created using a non-harmonic tone complex with 202 pure-tone components with 33.333 carriers per octave (CPO) from 100 Hz to 6400 Hz [16]. Using a 3AFC format, participants are presented with two reference stimuli at 20 RPO and a target stimulus at a different ripple density. Performance is measured as the highest ripple density the participant can distinguish from the 20 RPO reference. Resnick et al. [18] found that performance scores were affected by the carrier density (33.333 CPO) used in the original test. Normal hearing participants performed better with a carrier density of 100 CPO but exhibited non-monotonic behavior with 33.333 CPO, including a drop in performance at 11 RPO and improvement at 16 RPO due to spectral aliasing caused by the Nyquist limit [18]. Narne et al. [19] hypothesized that amplitude fluctuations due to cochlear filtering around or above 6400 Hz may act as an unintentional cue, explaining why the SMRT yields higher thresholds (7.1 RPO) in normal hearing listeners compared to a spectral ripple test (5.5 RPO). However, this spectral ripple test differed from the spectral ripple test by Won et al. in several characteristics, such as peak-to-valley ratio, a different frequency range, and filtering. For CI users, Lawler et al. [20] reported an average threshold of 3.1 RPO in the SMRT. However, both tests have shown correlations with speech understanding and speech recognition in noise, highlighting their importance despite their limitations [10,20,21].

1.3. Modeling Approach

This paper evaluates whether models of normal and electric hearing can replicate the unintentional cues and outcomes observed in the spectral ripple test and SMRT, such as aliasing, the benefits of current steering, and reduced ripple resolution in electric hearing. If successful, this would validate the use of model-based analyses to assess psychophysical tests without patient involvement. For normal hearing, we used the model from Bruce et al. [22], which simulates auditory signal transmission via sensory hair cells to auditory nerve fibers. This phenomenological model can simulate spiking patterns at characteristic frequencies in response to auditory inputs. For electric hearing, we introduce a new advanced modeling pipeline developed at our department that combines biophysical and phenomenological models [23,24,25,26,27,28,29]. This pipeline includes a speech processor to simulate spiking patterns in response to auditory inputs for a cochlear implant with an electrode array. More details on the models are provided in the Materials and Methods section. These models provide insights into what is perceived in normal hearing and what is conveyed by speech processors in electric hearing, enabling a deeper understanding of psychophysical test outcomes and paving the way for more reliable test designs.

2. Materials and Methods

A schematic overview of both models and their components is shown in Figure 1. The electric hearing pipeline consists of a model for a speech processor and a cochlear model. The speech processor translates a sound wave into an electrodogram, which contains the pulse amplitudes per electrode and is in steps of the pulse width. The implanted cochlea model, which consists of three separate models, receives the electrodogram as input and can calculate the spike times per fiber in response to this input, as will be described in Section 2.1. In comparison, the normal hearing model processes the sound wave directly without preprocessing. It is an elaborate and well-validated model with intricate processing of each step of normal hearing, and the steps within the model are described in Section 2.2. An overview of the settings of both models is provided in

Table 1. Model settings of the electric hearing model and the normal hearing model. Both models had the same frequency range (250–8054 Hz), the same number of fibers (2416 fibers), the same number of trials (5 trials), and the same window size for moving average filtering (33, see Section 2.4.1). SPL: sound pressure level. FS: full scale. RMS: root mean square.

Parameter	Electric Hearing	Normal Hearing
Number of fibers per frequency	1	50
Loudness	65 dB SPL (FS 111.6 dB)	65 dB RMS SPL
Spontaneous rate [spikes/s]	10	0.1 (n = 10), 4 (n = 10), 70 (n = 30)

.

2.1. Electric Hearing

In this paper, we propose a new extensive electric hearing pipeline that models how sound is processed by the fibers of the cochlear nerve of a CI user. The pipeline here models a speech coding strategy resembling the Fidelity 120 strategy of Advanced Bionics, which is described in Section 2.1.1. The implanted cochlea model uses the output of the speech processor to model the fiber responses to the input and is presented in Section 2.1.2. Lastly, an interpretation step in the form of a new frequency allocation is added to the output of the fibers to evaluate the perceived spectral content, which is explained in Section 2.1.3.

2.1.1. Speech Processor

The speech processor is taken from the open-source Advanced Bionics Generic-Python-Toolbox and generates pulse trains in response to sound waves. This is code for the spectral resolution (SpecRes) strategy (Figure 2), a research version of the commercial HiRes with Fidelity 120 (F120) strategy [29]. The input level was rescaled to 65 dB sound pressure level (SPL) with the full scale set to 111.6 dB. The range of the CI analysis bands is 306–8054 Hz.

The sound is preprocessed with a pre-emphasis filter, to attenuate lower frequencies and thus balance the frequency spectrum, and an automatic gain control, to maintain a suitable signal amplitude at its output, despite variation of the signal amplitude at the input. The filter bank consists of a fast Fourier transform (FFT), which reflects the frequency content per time frame or window, and is divided into 15 frequency bands. The output of the filter bank is used to assess the envelope of the input and to estimate the frequency per frequency band. The envelope information is used by a noise reduction algorithm, which is comparable to the ClearVoice algorithm of Advanced Bionics, in which the speech energy and noise energy are estimated, and a correction in the form of gains is applied. In parallel, the frequency estimate information is used to determine the carrier frequency and the frequency location within the band (for current steering, explained in the following paragraph). Lastly, these two parallel lines of information are used to map frequency content to the T- and M-levels to produce an electrodogram.

This speech coding strategy uses current steering, a method to enhance the frequency resolution in cochlear implants, where two neighboring electrodes are stimulated simultaneously to elicit neural activation between these two physical contacts. Sixteen electrodes are used in this speech coding strategy, resulting in 15 pairs of electrodes called analysis bands or channels (see

Table 2. The 15 analysis bands of the Fidelity 120 speech coding strategy are divided over 16 electrodes. For each analysis band, the lower and upper frequency edges (corner frequencies) are listed in Hz. The upper edge of each band is the lower edge of the next band.

Analysis Band	Lower Edge	Upper Edge
1	306	442
2	442	578
3	578	646
4	646	782
5	782	918
6	918	1054
7	1054	1257
8	1257	1529
9	1529	1801
10	1801	2141
11	2141	2549
12	2549	3025
13	3025	3568
14	3568	4248
15	4248	8054

for the frequency assignment). Thus, each analysis band has two assigned electrodes: the lower corner frequency is assigned to the lower electrode of the analysis band, and the upper electrode is assigned to the upper corner frequency or edge of the analysis band. By varying the weight ($\alpha$) of the current in the electrodes in steps of 0.125, the peak location can be shifted. This results in nine possible sets of weight combinations and peak locations per frequency band per time frame. With the weight set to ${\alpha }_1=0.5$ and ${\alpha }_2=0.5$ for electrodes 1 and 2, respectively (i.e., equal amplitudes), the activation peak resides in the middle of electrodes 1 and 2. For example, a signal of 442 Hz (see Table 2) is assigned to electrode 2 (with ${\alpha }_2$ = 1 and ${\alpha }_3$ = 0), and a 578 Hz signal is assigned to electrode 3 (${\alpha }_3$ = 1 and ${\alpha }_4$ = 0). For a more detailed explanation on current steering and the speech coding strategy, we refer to [29].

By setting the weights for all 15 channels to 0.5, we mimicked a different speech coding strategy without current steering, meaning all 15 peaks were placed in the middle of the electrodes. This is similar to the HiResolution strategy used by de Jong et al. [30], dubbed the HiRes FFT strategy. However, they used 16 channels, with the peaks situated directly at the electrodes by setting the $\alpha$ values to 0.

The frequency content per channel, meaning the energy estimated by the speech coding strategy per analysis band, was used to assess how much information can be conveyed by the speech coding strategy, as was done by Winn and O’Brien [14]. This assessment used the frequency content after the addition of the noise reduction gains to the envelope calculation per band (see red block in Figure 2). The T-levels and M-levels were set to those of the implanted cochlea model (described in Section 2.1.2).

2.1.2. The Implanted Cochlea Model

The implanted cochlea model consists of a volume conduction model, an active nerve fiber (ANF) model, and a stochastic nerve model (see the top of Figure 1 for an overview). The 3D volume conduction model calculates the potential distribution along the modeled nerve fibers in the implanted cochlea per electrode contact [25,26]. The potential distribution is then used by the ANF model to compute a deterministic threshold for each electrode pair [23,27]. The cochlear geometry used in the ANF model is based on µCT imaging data from human temporal bones, and realistic neural trajectories are defined according to histological data. Next, the phenomenological adaptive stochastic (PHAST+) model takes the deterministic threshold and adds stochasticity and temporal effects (refractoriness, accommodation, and adaptation) to simulate the resulting spike train per auditory nerve fiber in response to the pulse train [24,28]. An overview of the included neural parameters is listed in

Table A1. The included neural parameters for the electric hearing pipeline. The parameters are listed with their abbreviation or variable as listed in the reference. SD: standard deviation.

Parameter	Value (SD)	Reference
Absolute refractory period (ARP)	0.0004 s (0.0001 s)	[24]
Relative refractory period (RRP)	0.0008 s (0.0005 s)	[24]
Relative spread (RS)	0.06 (0.04)	[24]
Accommodation rate (${\theta }_a$)	0.014	[28]
Adaptation rate (${\theta }_s$)	19.996	[28]
Accommodation amplitude (${\omega }_a$)	0.072	[28]
Adaptation amplitude (${\omega }_s$)	7.142	[28]
Fiber spacing	9.6 μm (0.46 μm)	[25]
Spontaneous rate (SR)	10 Hz

in the Appendix A. The model can process the response of 3200 fibers distributed between 2.56 mm and 33.2 mm from the round window. However, for our purposes, we limited it to the fibers stimulated in proximity to the implanted electrode array, resulting in 2416 fibers. The ANF model modeled the thresholds, T-levels, and M-levels of the HiFocus MS electrode array (Advanced Bionics) in a mid-scalar position. The insertion angle of the most basal electrode contact was 27 degrees from the round window, and for the most apical electrode contact, 420 degrees from the round window, modeled following the mean angular insertion depth in van der Jagt et al. [31]. The T-levels and M-levels were estimated based on 0.5 mm and 3.5 mm excitation along the basilar membrane, respectively, per electrode. A known shortcoming of the ANF model is that the thresholds are approximately an order of magnitude too high compared to physiological values. To avoid affecting the speech coding strategy with this discrepancy, the T-levels and M-levels used by the speech coding strategy were divided by 3 to be in the same ballpark as physiological levels, and the amplitudes of the pulses of the electrodogram were multiplied by 3 to use them as an input for the implanted cochlea model. We used five trials per stimulus to get an averaged response, meaning the neural response was calculated for a single stimulus five times and averaged over these five runs.

2.1.3. Fiber Frequency Allocation

Generally, the Greenwood frequencies are used based on the basilar membrane position. However, the consensus is that cochlear implant (CI) users adapt over time to the (frequency-to-place) mapping of the implant [7,32,33,34]. Here, we have assumed that neural adaptation to the new CI-based frequency distribution has occurred, and we have accordingly created a new, so-called ‘learnt’, fiber frequency allocation (see Figure 3A,B). Our frequency allocation approach is similar to the approach of de Nobel et al. [35]. Although this adaptation is presumed to occur higher up than the auditory nerve, we allocated these new frequencies to the fibers for interpretation of the neural processing, which greatly simplified the comparison between the two models. For each electrode, the fiber with the lowest threshold (see Figure 3) was assigned the corner frequency of that electrode (see Section 2.1.1). The fibers between these highest responding fibers were assigned frequencies log-spaced between the edges of the frequency bands (see Figure 3D). A different approach was applied to the first and last electrodes. The fibers situated just outside of the array, with a threshold up to M-level of these contacts, were included. For the first analysis band and fiber section, the fibers were assigned frequencies logarithmically spaced between 250 Hz and 442 Hz. A lower frequency edge was chosen to include frequencies to which the speech coding strategy might respond due to spectral leakage, due to the imperfect resolution of the FFT filter banks. This frequency was chosen heuristically by providing the speech coding strategy with low-frequency (below 306 Hz) pure sine stimuli and assessing when the electric output increased. For the last analysis band and fiber section, the fibers were assigned frequencies logarithmically spaced between 4248 Hz and 8054 Hz.

2.2. Normal Hearing Model

The phenomenological model from Bruce et al. [22] was used to simulate the response of single fibers in normal hearing. This model captures the transmission of auditory signals from the ear canal, via the middle ear and basilar membrane, to the sensory hair cells in the inner ear, to the auditory nerve fibers, simulating spiking patterns in response to specific auditory inputs across chosen characteristic frequencies. The model includes cochlear filtering, which replicates the frequency-selective response of the basilar membrane using a bank of bandpass filters that mirror the cochlea’s tonotopic organization. The next stage involves inner hair cell processing, where basilar membrane motion is transformed into receptor potentials to reflect the dynamic range of human hearing. Synaptic transmission is modeled to simulate neurotransmitter release from the inner hair cells to auditory nerve fibers, including short-term adaptation dynamics that influence nerve responses. The auditory nerve fiber response is then captured, simulating the spiking behavior of fibers with stochastic elements to reflect the natural variability in spike timing. The model represents different types of nerve fibers with low, medium, and high spontaneous rates to account for the diversity in auditory nerve responses. The output per characteristic frequency was created using the model’s default settings: 10 low-spontaneous, 10 medium-spontaneous, and 30 high-spontaneous fibers. The spikes of these fibers were summed and averaged over five trials. The default setting of variable fractional Gaussian noise was used. The basilar membrane tuning for humans was based on Shera et al. [36]. The approximate implementation of the power-law functions in the synapse model was chosen. The input sound wave of the normal hearing model was set to 65 dB root mean square (RMS) SPL. The characteristic frequencies of the fibers were set to the same 2416 frequencies as used in the electric hearing pipeline, ranging from 250 Hz to 8054 Hz.

2.3. Sounds

2.3.1. Spectral Ripple Test

The first test used the ripple sound files of Won et al. [10]. In this test, the listener has to distinguish between two noisy stimuli with the same ripple density; the ripples in the frequency content are shifted, creating standard and two inverted stimuli (i1 and i2, see Figure 4). The inverted stimuli i1 and i2 have the same pattern in the power spectral density but have slightly different randomly included frequencies. The stimulus duration was 0.5 s, and the stimuli were ramped with 150 ms rise/fall times. Frequencies from 100 Hz up to 5 kHz are included. The test becomes more difficult with increasing ripple density as the peaks of the ripples fall closer together. Thus, this test makes use of the ability to discriminate between frequencies along the basilar membrane. The original ripple test contained 14 different densities, which differed by ratios of 1.414 (0.125, 0.176, 0.250, 0.354, 0.500, 0.707, 1.000, 1.414, 2.000, 2.828, 4.000, 5.657, 8.000, and 11.314 RPO). The peak-to-valley ratio of these stimuli was 30 dB, and they were filtered with a long-term speech-shaped filter [37]. For each ripple density, 30 stimuli with different starting phases in the spectral domain are possible. All figures are created with the first phase (indicated with ‘_1’ in the sound filename) and comparing s and i1 (not to overcrowd the figures), except for the spectral ripple threshold determination (see Section 2.5).

2.3.2. SMRT

For the second test, the SMRT stimuli (see Figure 5) described in Aronoff and Landsberger [16] were recreated in Matlab (2023a) with a sampling rate of 44.1 kHz. The amplitude modulations A applied to amplitude of the pure tone carrier P to derive stimulus S with

$$S\left(t\right)=\sum _{i=1}^C{P}_i\left(t\right)·{10}^{{\displaystyle \frac{A(i,t)}{20}}},$$

(1)

where t is time, i is the index of the pure tone, with the number of carriers C at 201 for the lower carrier density (33.333 CPO) and 602 for the higher carrier density (100 CPO). The amplitude modulation A is calculated as follows:

$$A(i,t)=\left|D·sin\right(0.03\pi Qi+R\pi t\left)\right|-D,$$

(2)

with a ripple depth D of 20 dB, a ripple repetition rate R (the number of repetitions of the ripple per second) was set to 5 Hz, the ripple density Q expressed in ripples-per-octave (RPO). The stimuli were generated using a complex of tones with 201 or 602 equal-amplitude pure-tone frequency components with 33.333 CPO or 100 CPO (indicated by red dots on the left-hand axis of Figure 5) of an octave from 100 Hz to 6500 Hz for the lower and higher carrier density, respectively. A higher density would mean a greater number of red dots. The stimuli had a duration of 0.5 s, and linear ramps of 0.1 s were added to the beginning and end of the sound.

2.4. Visualization

2.4.1. Spectrum

For the spectral ripple sounds [10], a spectrum view was used to indicate the spectral content of the sound. Three types of spectrum views were used: acoustic, electric, and neural. The latter portrays the amount of neural activity, the other spectra the power in a frequency bin over the entire stimulus duration. All spectrum views were min–max-normalized for better comparison.

The acoustic spectrum shows the normalized power spectral density (Figure 4 and Figure 6A–D). In contrast to Winn and O’Brien’s method, the actual sound was used to calculate the electric spectrum and neural spectrum, not an ideal spectrum [14]. As a result, the ripples varied in height due to the sounds being filtered with a long-term speech-shaped filter [10,37]. For better visualization of the basilar membrane response, the same pre-emphasis filter in the speech processor was applied to the acoustic spectra.

The electric spectrum (Figure 6E–H) shows the normalized output (the perceived power per analysis band) of the speech coding strategy after the noise reduction step, before mapping, to evaluate the frequency content per frequency band (see red block in Figure 2).

For the neural spectrum, the number of spikes generated by the neural models over the stimulus in a frequency bin of the neural models was used. For all of the following neural spectrum figures in the Results (Figure 7 and Figure 8), the y-axis indicates the normalized response over all fiber responses, and the x-axis indicates the mapped frequency of the fiber. For the normal hearing output, this is the characteristic frequency. Individual fiber responses varied due to stochastic properties and were rather large in number, resulting in a somewhat chaotic spectrum. Therefore, in addition, the frequency response of the individual fibers was also spatially low-pass filtered to show the general response along the basilar membrane with a symmetric moving average filter.

2.4.2. Neurogram

To assess the presence of aliasing in the temporal domain, neurograms were chosen as the method for visualization (Figures 10 and 11). A neurogram is made by stacking the post-stimulus time histograms (PSTHs) of all fiber responses. The PSTHs were created with a bin size of 5 ms. Neurograms portray the neural response of the auditory nerve in terms of the neural spiking intensity of multiple auditory nerve fibers as a function of time and location on the basilar membrane. As auditory nerve fibers have a topological order in terms of characteristic frequency, neurograms look similar to spectrograms. The colors of the bins in the neurograms indicate the spike rate per frequency and time bin. Brighter colors reflect a higher spike rate.

2.5. Spectral-Ripple Threshold

We modeled the neural spectrum for multiple starting phases per ripple density in addition to multiple trials per stimulus. This provided us with enough trials to calculate $d^{\prime }$ for the standard and the inverted stimulus at each ripple density. We calculated $d^{\prime }$ following the method of [38,39], which assumes unequal variances, using

$$d^{\prime }={\displaystyle \frac{\left|{\mu }_s-{\mu }_i\right|}{\sqrt{\frac{1}{2}\left({\sigma }_s^2+{\sigma }_i^2\right)}}},$$

(3)

where ${\mu }_s$ is the mean of the neural spectrum for the standard stimulus, ${\mu }_i$ is the mean for the inverted stimulus, ${\sigma }_s$ is the standard deviation of the neural spectrum in response to the standard ripple, and ${\sigma }_i$ is the standard deviation of the inverted ripple. For the inverted ripple, two versions (i1 and i2) are available per phase, providing three options for a 3-alternative forced-choice test. The average $d^{\prime }$ of both inverted stimuli ($d_{si1}^{\prime }$ and $d_{si2}^{\prime }$) was used to simulate the proportion of correct $P_c$ to plot the psychometric curve for a 3AFC test:

$$P_c\left(d^{\prime }\right)={\int }_{-\infty }^{\infty }\Phi {\left(x\right)}^2\phi \left(x-d^{\prime }\right)dx,$$

(4)

where $\Phi \left(x\right)$ is the cumulative distribution function of a standard normal distribution and $\phi \left(x\right)$ is the probability density function of a standard normal distribution [40,41,42,43]. We estimate the spectral-ripple threshold T in this 3AFC test:

$$T=\gamma +{\displaystyle \frac{1-\gamma }{2}},$$

(5)

with the guessing rate $\gamma$ = ${\displaystyle \frac{1}{3}}$, resulting in a threshold at a proportion of correct $P_c$ = 0.667 or 66.67% correct [43]. This assessment was performed on (moving average filtered) neural spectra.

3. Results

3.1. Spectral Ripple Test

3.1.1. Acoustic and Electric Spectra

Winn and O’Brien [14] looked at the acoustic and electric content for the spectral ripples from Won et al. [10]: the frequency spectrum of the sound and the frequency content that is transferred through the speech coding strategy. Their results were recreated with our model in Figure 6. In the top row, the acoustic content of four different spectral ripple stimuli, with standard and inverted versions, is shown in order of increasing ripple density from left to right: 0.5 RPO, 1.1414 RPO, 2.0 RPO, and 4.0 RPO. In the acoustic spectrum (Figure 6A–D), the standard (blue) and inverted (red) peaks are easily distinguished up to 4.0 RPO. From the grey vertical lines, indicating the frequency band edges of the speech processor, it is clear that for electric hearing, the standard and inverted peaks are often in the same analysis band at 4.0 RPO. The bottom row shows the corresponding electric spectrum (i.e., the spectral information conveyed by the speech coding strategy per analysis band). Similarly, the spectral content of the standard (blue) and inverted (red) stimuli is shown in order of increasing ripple density. Spectral content below 340 Hz was not transmitted through the speech coding strategy, as the lowest frequency band ranged from 306 Hz to 442 Hz. Thus, no content was shown in the electric spectrum. At low ripple density (0.5 RPO, Figure 6E), the speech coding strategy could accurately follow the ripples in the spectral domain, as a wavy pattern was visible in the spectrum. With greater ripple density (1.414 and 2.0 RPO, Figure 6F,G), the CI processor could still follow the ripple as the peaks alternated between the edges of the frequency band. At 4.0 RPO (Figure 6H), the frequency content was less discernible between the standard and inverted ripples due to multiple peaks being within the same analysis band, resulting in similar content for the standard and the inverted stimulus. Moreover, a descending ripple or wave pattern in the electric spectrum of (Figure 6H), similar to aliasing, was seen in the inverted content and more clearly in the standard content with peaks situated at the 442–578 Hz, 646–782 Hz, 1257–1529 Hz, 2549–3025 Hz. In other words, the ripple density of the ripple pattern in the electric spectrum in Figure 6H was lower than the ripple density in the acoustic spectrum (Figure 6D). However, some bands differed between the standard and inverted content, such as between 306–442 Hz and 1801–2141 Hz, indicating that some CI users may be able to use this loudness cue to distinguish the stimuli.

3.1.2. Neural Spectra

Figure 7 shows the neural activation for normal hearing and the neural activation with the F120 speech coding strategy for the standard and inverted stimuli in ascending ripple density per row. The average $d^{\prime }$ values over all phases are listed in the caption per ripple density. Similar to the ripples in the normalized power spectral density in Figure 7A–D, the neural spectrum shows ripples along the basilar membrane. For normal hearing, the peaks of the standard and inverted stimuli were discernible at every depicted ripple density. With increasing ripple density, the peaks became slimmer but were still specific to the location on the basilar membrane, though the valleys became shallower with increasing ripple density due to the peaks being closer to each other. This alludes to increased difficulty in discrimination. For electric hearing, the discernibility between the peaks of the standard and inverted stimuli rapidly decreased with increasing ripple density. At $0.5$ RPO (Figure 7B), the peaks were situated at different locations on the basilar membrane. At $2.0$ RPO (Figure 7F), this also appeared to be the case, but the peaks were less distinctly located than what was observed at the lower ripple density. At 4.0 RPO (Figure 7H), the peaks of the standard and inverted ripples coincided at the same location of the basilar membrane for electric hearing, which was not the case for normal hearing. In electric hearing, the peaks in the highest frequency band (4–8 kHz) did not coincide, reflecting that the upper standard peak is higher than the upper inverted peak (Figure 6D). In the normal hearing response, a clear decrease in neural activation occurred after 5 kHz, the upper limit of the frequency content of the spectral ripple; this was not the case for electric hearing. For normal hearing, the activation was relatively strong over all frequencies up to 5 kHz. For electric hearing, the drop in activation is not as distinct as with normal hearing. The spatial profile of activation (derived by spatially filtering the individual fiber responses), shown as thick lines in the spectra for normal hearing, consistently concurred with the peaks of the bins. In contrast, for electric hearing, the thick line fell lower than the peaks of the individual fibers, a lower height than for normal hearing. This indicates that the fiber response rates are more variable for electric hearing.

Figure 8 shows the neural activation with the F120 speech coding strategy (Figure 8A) and without current steering (Figure 8B) for the standard and inverted stimuli with 2.828 RPO. The average $d^{\prime }$ values over all phases are listed in the caption for both strategies. The summed squared difference of the individual fiber responses between the inverted and standard stimuli was larger with current steering (25.5) than without current steering (6.2). For the filtered response (thick line), the summed squared difference was 21.6 with current steering and 5.4 without current steering. Apparently, current steering offers a benefit in discernibility; for example, the peaks within the ranges of 782–1054 Hz (channels 5 and 6), 1257–1801 Hz (channels 8 and 9), 2549–3568 Hz (channels 12 and 13), and 4248–8054 Hz (channel 15) are separable.

3.1.3. Spectral Ripple Test Threshold

The filtered neural spectra were used to determine the spectral ripple threshold by simulating the psychometric curves. The psychometric curves, shown in Figure 9, have been fitted with a logistic function on filtered spectra (the thick lines in Figure 7 and Figure 8) of normal hearing ($R^2=0.999$), electric hearing ($R^2=0.995$), and electric hearing without current steering ($R^2=0.995$). The results per phase and inverted stimulus are shown as smaller dots. Around the threshold, there is a greater spread in performance over the phases. Per ripple density, two large dots are plotted from each average$d^{\prime }$ of 30 phases; however, they often concur. For normal hearing, the threshold is not reached within the tested sample of ripple densities. The interpolated spectral ripple threshold from the fitted psychometric curve with F120 and current steering (T = 3.7 RPO) is higher than for F120 without current steering (T = 2.2 RPO).

3.2. SMRT

For the four SMRT spectrograms in Figure 1 from Resnick et al. [18] neurograms were simulated with the same ripple density (4.0 and 16.0 RPO) and both carrier densities (33.333 CPO and 100 CPO). The neurograms for normal hearing are shown in Figure 10. At 4.0 RPO, the ripples were visible at both carrier densities (Figure 10A,B). However, at the carrier density of 33.333 CPO with 16.0 RPO (Figure 10C), ripples were visible in the opposite direction (bottom left to top right), similar to Resnick’s spectrogram for this situation. With the carrier density of 100 CPO, the ripples could no longer be discerned at 16.0 RPO (Figure 10D). Although the SMRT stimuli were created with frequencies up to 6500 Hz, some neural activation was seen above this frequency (6500 Hz to 8000 Hz) in synchrony with the activation at 6500 Hz. The darker areas at the beginning and end of the stimuli are due to the ramps. When focusing on a slimmer frequency band, such as from 3500 Hz to 4500 Hz (similar to looking at a critical band), these ripples were visible as amplitude modulations in the neural activation. In some regions, vertical lines can be discerned. These lines are also present in the acoustic stimulus (see Figure 5).

The neurograms created with the electric hearing pipeline had a lower ripple density in order to be in the perceptible range of CI users. Figure 11 shows the neurograms with 1.0 RPO (Figure 11A), 2.0 RPO (Figure 11B), and 3.0 RPO (Figure 11C). At 1.0 RPO and 2.0 RPO, the ripples were visible and had a smooth course due to current steering. At 3.0 RPO, the ripples were difficult to discern. They were vaguely visible between 3 kHz and 4 kHz. Considering that 6500 Hz falls within the highest frequency band of channel 15 (4248–8054 Hz), the current will be allocated to electrode 16, inducing neural excitation in these high frequencies.

4. Discussion

Here, we have studied how the stimuli of the spectral ripple test and SMRT are processed with models of normal hearing and hearing with a CI. The models used here have demonstrated the anomalies in the clinical results of these tests.

Based on the information conveyed through the speech processor (Figure 6), the original spectral ripple test by Won et al. [10] is limited in the transmission of peaks and valleys for higher ripple densities. This aligns with the findings and hypotheses of earlier studies [11,12,13]. Moreover, unlike in the acoustic case, for CIs, spectral aliasing is visible at 4.0 RPO due to the limited spectral resolution of the filter bands, in the form of a slower wave pattern of the electric content. In other words, the CI was unable to follow the higher ripple density of the spectral ripples, resulting in the presentation of a ripple of lower ripple density. This result corroborates the theory postulated by Winn and O’Brien [14] that the speech processor cannot follow above 2.0 RPO, which also turns out to be the visible limit in our results. Compared to Winn and O’Brien [14], we used a more realistic method to portray the electric content; they used an idealized approach with ideal filters as square windows on an ideal ripple spectrum, whereas we used the true stimuli and the research version of a speech processor. Moreover, they simulated the pattern of excitation with a 2 dB/mm roll-off, whereas we included an implanted cochlea model with 3D volume conduction. This limitation of 2 RPO is also seen in the neural activation in Figure 7, where the peaks are no longer distinguishable at 4.0 RPO. However, the aliasing is not visible in the neural activation. The spectrum results are congruent with the performances of CI users and normal hearing subjects. Won et al. [10] found an average ripple threshold of 1.73 RPO in CI users, with a range of 0.6 to 4.87 RPO. The majority of CI users in this study used the ACE (Advanced Combination Encoder) strategy and the remainder used SPEAK (spectral-peak speech coding strategy), CIS (Continuous Interleaved Sampling), or HiRes [10]. In a similar paradigm, Henry et al. [44] found a spectral ripple detection average of 1.77 RPO for CI users and 4.84 RPO for normal hearing participants. However, these stimuli were also slightly different from the stimuli of Won et al. [10] in terms of spectral shape (sinusoidal shape on a linear amplitude axis versus on a logarithmic axis). In another similar paradigm, Anderson et al. [11] found an average ripple discrimination threshold of 1.68 RPO (range of 0.41 to 4.27 RPO), but with cutoff frequencies of the noise pass band at 350 and 5600 Hz and a stimulus duration of 400 ms. Drennan et al. [45] reported a higher average ripple discrimination threshold of 2.60 RPO with a range of 0.63 to $7.03$ RPO over their CI users. In this study, a more diverse selection of strategies was included: ACE, CIS, SPEAK, HiRes, HiRes with F120, SAS (Simultaneous Analogue Stimulation), FSP (Fine Structure Processing), and CCIS (conditioned CIS). The electric hearing pipeline has also shown the benefit of the F120 strategy in the case of spectral ripples, as the peaks at $2.828$ RPO are better discriminable than with current steering turned off (Figure 8). Similarly, this improved discrimination was visible in our psychometric curves derived from the filtered neural spectra (Figure 9). Drennan et al. [15] had a similar finding when they compared the performance of the F120 strategy and the HiRes strategy in the spectral ripple test. The HiRes strategy employs real-time filtering with 16 logarithmically spaced bandpass filters and electrodes, whereas the F120 strategy uses an FFT filter bank and current steering over 15 frequency bands between 16 electrodes. They found a mean threshold of 3.42 RPO for the F120 strategy and 2.31 RPO for the HiRes strategy. Without any optimization or additional parameters, our results are in the same ballpark, with 3.7 RPO for F120 and $2.2$ RPO for F120 without current steering. However, the strategy without current steering is not the same strategy as ours, as it does not make use of FFT and includes 16 electrodes. Our strategy without current steering is based on the HiRes FFT strategy used by de Jong et al. [30] and setting the weights to 0.5 with 15 frequency bands, resulting in 15 unique peak positions. The channel-specific temporal envelopes can be more faithfully represented in the original HiRes strategy, but a spectrum is unlikely to give us information on this difference. Nonetheless, de Jong et al. [30] found no difference in SMRT performance between the original HiRes strategy with a bandpass filter bank and their HiRes FFT strategy.

These tests and models have given insights into the differences in the spectral resolution of normal hearing and electric hearing. The spectral resolution, in terms of ripple visibility along the spectrum or ripple visibility in the neurograms, is discernible up to higher ripple densities in normal hearing. Especially above 4000 Hz, greater discriminative power was discerned for normal hearing. This analysis band in the F120 strategy spans the largest frequency range and thus provides the least frequency resolution with current steering. In the neural spectrum for the spectral ripples, a difference in smoothness is similarly evident. The ripples in the electric hearing spectrum are jagged compared with those in the normal hearing spectrum. This is also visible in the filtered response, which varied in distribution. The threshold profiles created for this array are of a similar choppy nature [25]. As postulated by Narne et al. [19], amplitude modulation cues at the outputs of auditory filtering centered at the upper edge of or above the frequency range may be the reason why the ripple threshold is higher for SMRT than the spectral ripple test they used. In our neurograms, amplitude modulations above the frequency limit of SMRT were also visible (Figure 10).

The models have provided an interesting visualization of the perception of SMRT stimuli. In the normal hearing neurograms, the ripples were visible at both carrier densities for 4.0 RPO (Figure 10A,B). Furthermore, the aliasing artefact was visible at 16.0 RPO with a carrier density of 33.333 CPO (Figure 10C), which is consistent with the spectrogram of Resnick et al. [18]. For electric hearing, the ripple discrimination in neurograms, in terms of ripple visibility, was reduced compared to normal hearing (Figure 11). The ripples were hardly visible at 3.0 RPO, which is similar to what has been found among CI users in our department [46], whereas with normal hearing, these ripples were still clearly visible at 4.0 RPO. We can infer from these neurograms that the current steering strategy is capable of a more precise distribution of the peak in this range with 1.0 RPO (e.g., between 0.2 s and 0.3 s), but not with greater ripple density. At 3.0 RPO, the ripples are barely visible, except for the range between, for instance, 3 kHz and 4 kHz. In this regard, another cue that has been overlooked in the SMRT is that the CI participant may selectively use the difference in amplitude modulation within one channel, instead of the full spectrum, to identify the odd one out [47]. To address this cue, a new spectral resolution test was developed by Archer-Boyd et al. [47], the Spectro-Temporal Ripple for Investigating Processor EffectivenesS (STRIPES). In this test, listeners must distinguish between complexes of sine downward and upward sweeps. The difficulty is increased by increasing the spectro-temporal density of the stimulus. In addition, noise bookends were added to avoid onset and offset cues. However, analysis of this psychophysical test is beyond the scope of this paper but will be assessed in future research.

Several limitations of this study should be addressed. Both neural models used in this paper have only been validated on single-fiber data, whereas the output here is extrapolated to the entire cochlea. However, the first two components of the implanted cochlea model (Figure 1) are based on the entire cochlea. In the configuration of the normal hearing model used here, the frequency resolution was relatively high, and we might have overestimated the resolution of the filters that reflect the basilar membrane response. Similarly, we may have overestimated the extent to which a CI user learns to adapt to the new frequency allocation of the array, or we may have misaligned the fibers to the frequency allocation. Moreover, we assumed that the information we can gather visually is similarly processed by or distinguishable to the ear and auditory cortex. The extraction of processed information would be more objective with the addition of an interpretation model, such as the one described by Hamacher [48], but the discrimination between ripple densities of the SMRT was not that straightforward because it did not yield a monotonic decrease in the model performance. In other words, a higher ripple density did not translate to a trend of more difficult discrimination, as was possible with the spectral ripple test shown here in the psychometric curves (Figure 9). Similarly, with the d’ implementation we were not able to find such a pattern for the modeled SMRT neurograms. Again, this implies that for such intricate signals, a more elaborate interpretation model is necessary to replicate cortical processing. Alternatively, a neural network approach could produce better results. Nonetheless, the models here corroborate clinical results, giving the impression that they work well.

The pipeline presented in this paper, with normal and electric hearing, offers a tool for assessing other psychophysical tests, such as STRIPES, or for future psychophysical tests before assessment in participants. With this pipeline, unintentional cues present in the stimuli can be filtered out in advance. In addition, this pipeline could be a tool for assessing future speech coding strategies. The current degree of temporal and spectral resolution of the information conveyed by speech coding strategies leaves room for improvement. The pipeline with both normal hearing and electric hearing allows for an easy first assessment of a new speech coding strategy ‘in silico’. Another purpose for this pipeline could be to evaluate the characteristics of the CI or the speech coding strategy, such as the FFT assignment or filter banks to electrodes. Using artificial intelligence, a speech coding strategy could be optimized based on the difference between the outputs of the two models. If the neural activation of electric hearing is as similar to that of normal hearing as possible, we hypothesize that the sound experienced by CI users will be improved and more similar to that of normal hearing.