A Soft Parameter Function Penalized Normalized Maximum Correntropy Criterion Algorithm for Sparse System Identiﬁcation

: A soft parameter function penalized normalized maximum correntropy criterion (SPF-NMCC) algorithm is proposed for sparse system identiﬁcation. The proposed SPF-NMCC algorithm is derived on the basis of the normalized adaptive ﬁlter theory, the maximum correntropy criterion (MCC) algorithm and zero-attracting techniques. A soft parameter function is incorporated into the cost function of the traditional normalized MCC (NMCC) algorithm to exploit the sparsity properties of the sparse signals. The proposed SPF-NMCC algorithm is mathematically derived in detail. As a result, the proposed SPF-NMCC algorithm can provide an efﬁcient zero attractor term to effectively attract the zero taps and near-zero coefﬁcients to zero, and, hence, it can speed up the convergence. Furthermore, the estimation behaviors are obtained by estimating a sparse system and a sparse acoustic echo channel. Computer simulation results indicate that the proposed SPF-NMCC algorithm can achieve a better performance in comparison with the MCC, NMCC, LMS (least mean square) algorithms and their zero attraction forms in terms of both convergence speed and steady-state performance.


Introduction
In nature, the impulse response of most unknown systems can be regarded as sparse, which consists of only a few dominant coefficients [1][2][3][4].The prior known sparse information can be used for improving the estimation performance in signal processing.Thus, sparse signal processing has been garnering significant attention in recent decades [1][2][3][4][5].In particular, the developed sparse signal processing techniques have been used in wireless communications, speech signal processing and imaging processing, which include compressed sensing (CS) [6][7][8] and sparse adaptive filtering .Although compressed sensing algorithms are useful for dealing with sparse signals, they need to construct measurement matrices with the limitation of restricted isometry property (RIP) [31].Thus, the computational complexity might be higher in comparison with sparse adaptive filtering algorithms [21][22][23].
Fueled by the demand of sparse signal processing, sparse adaptive filter algorithms have been widely studied for sparse system identification, echo cancellation and channel estimation .
and zero attracting techniques.The proposed SPF-NMCC algorithm is realized by normalizing the MCC algorithm to form a normalized MCC (NMCC) algorithm that is similar to the NLMS algorithm.Then, a soft parameter function is included into the cost function of the NMCC algorithm to give a desired zero attractor term in the update equation of the SPF-NMCC algorithm.The proposed SPF-NMCC algorithm is derived in detail based on the Lagrange multiplier method, and its performance is investigated for estimating a sparse system and an echo channel.The results obtained from the computer simulations show that the proposed SPF-NMCC algorithm is better than the traditional MCC, ZA-MCC, and RZA-MCC algorithms in terms of the convergence speed rate and the steady-state performance.
The rest of this paper is structured as follows.In Section 2, the traditional MCC algorithm and its sparse forms are reviewed and discussed within the system identification framework.In Section 3, the proposed SPF-NMCC algorithm is described mathematically in the framework of sparse system identification, which is based on the normalized adaptive filtering theory, MCC algorithm and zero attracting theories.In Section 4, the estimation behaviors of the proposed SPF-NMCC algorithm are well investigated in the context of sparse system identifications.Finally, a conclusion of this work is given in Section 5.

Traditional MCC Algorithm
We review the traditional MCC algorithm in the context of sparse system identification.Herein, we are interested in the adaptive estimation vector ŵ(n) ∈ R N , which should be similar to the unknown system w o (n) ∈ R N , by reducing the difference between the desired signal d(n) and the estimation output y(n).Here, and where T , and v(n) is an additional Gaussian white noise signal.In general, the instantaneous estimation error can be written as where e(n) is the difference between the reference signal and the adaptive filter output.In practical engineering, w o (n) may be a sparse multi-path channel, an acoustic channel of a room, an underwater communication channel or a network echo channel, which should be identified by obtaining the echo cancellation.Similarly to the NLMS algorithm [40], the minimization of the difference of the least Euclidian norm is used to get the update equation of the MCC algorithm [2,44,45] under a constraint.
Here, the least Euclidian norm of the difference is where α is a very small constant.Then, we minimize According to the previous studies, we consider the update Equations in [2,44,45].Then, we have Thus, the MCC algorithm is to solve the following problem [2,44,45] where • 2 is the Euclidean norm of a vector, and ξ = χ x(n) 2 .Here, χ acts as a step-size.
In this paper, we use the Lagrange multiplier method to find the solution of Equation ( 8).Thus, the cost function of the MCC algorithm can be written as [44] where λ denotes the Lagrange multiplier.In order to minimize Equation ( 9), we calculate its gradient with respect to ŵ(n + 1).Then, we have From Equation (10), we have ŵ(n + 1) = ŵ(n) + λx(n), (11) and we can get Lagrange multiplier, which is written as By substituting Equation (12) into Equation (11), we have We can see that the MCC algorithm employs an exponential term to eliminate the large errors, rendering it robust against impulsive-like noises.

Zero Attracting Techniques
As we know, the zero attracting technique has been used to make use of the sparse property of in-nature signals such as wireless multi-path channels and echo channels [9].Recently, zero attracting methods have been used for exploiting the sparse MCC algorithms [49], resulting in two sparse MCC algorithms named as zero attracting MCC (ZA-MCC) and reweighted ZA-MCC (RZA-MCC) algorithms.The ZA-MCC algorithm is realized by using an l 1 -norm to modify the cost function of the traditional MCC algorithm, and, hence, the ZA-MCC algorithm is to solve the following problem [49] min 1  2 ŵ(n By using the Lagrange multiplier method to find the solution of Equation ( 14), we write the cost function as [49] J where γ ZA is a regularization parameter which is greater than zero, and • 1 represents the l 1 -norm.Then, the updated equation of the ZA-MCC algorithm can be expressed as where ρ ZA is a zero attracting controlling parameter that is used for balancing the zero attraction ability, and sgn [x] is an element-wise sign operator, which is defined as Comparing with the MCC algorithm, it is found that there is an additional term −ρ ZA sgn [ ŵ(n)] which is used to provide a zero attracting effect to make the ZA-MCC algorithm converge more quickly [9,49].Here, −ρ ZA sgn [ ŵ(n)] is also defined as a zero attractor in the ZA algorithms [9,22,49].Similar to the previously proposed zero-attracting-based sparse LMS algorithms, the ZA-MCC algorithm can be used for dealing with sparse system identification.After that, the RZA-MCC algorithm has been proposed by using a sum-log function instead of the l 1 -norm to further enhance the ZA-MCC algorithm for handling less sparse systems [49].The RZA-MCC provides a reweighting step-size in its zero attractor to selectively give penalties on the zero taps rather than uniformly exert penalties on all the taps.

Proposed Sparse SPF-NMCC Algorithm
Although the ZA-MCC and RZA-MCC algorithms can well utilize the sparse property of the in-nature systems, they might be affected by scaling of the input signals.As is known to us, normalized adaptive filtering can give resistance to the effects on this scaling, and the zero attracting technique is an effective method to estimate sparse systems.Motivated by the advantages of the normalized adaptive filters and zero attracting techniques, we propose a SPF-NMCC algorithm to further exploit sparseness characteristics for sparse system identification.
Similar to the sparse MCC algorithms, a soft parameter function is used to exploit the sparsity characteristics of these systems, which is defined as where β > 0. The performance of the soft parameter function is shown in Figure 1.It can be seen that the soft parameter function is close to l 0 -norm when β is very large, while it can be regarded as a l 1 -norm for small β.We can choose a large β to approximately implement the l 0 -norm penalty, and we can use small β to construct l 1 -norm penalty to exploit the sparsity of existing sparse systems.Then, we integrate this soft parameter function into the cost function of an NMCC algorithm to exploit the sparse characteristics of in-nature systems such as echo channels.Thus, the proposed SPF-NMCC algorithm is to find out a solution of the following problem where γ SPF is a regularization parameter that can be tuned to give a tradeoff between sparsity and estimation performance.Then, the cost function of the SPF-NMCC algorithm can be written as By using Lagrange multiplier method on Equation (20), we have and Then, we have and where and its vector form is Left multiplying by x T (n) on the Equation ( 23) and moving ŵ(n + 1) to the left side, we can obtain Combining with Equations ( 24) and ( 27), If we take Equations ( 23) and ( 28) into consideration and consider ŵ(n + 1) = ŵ(n) when the iteration is stable, we can obtain Discarding the term γ SPF since it is very small, we can get the update equation of the SPF-NMCC algorithm, which is given by where µ SPF = ξ acts as a step-size.It is observed that Equation (30) represents a unified form of the sparse MCC algorithm.The previously reported sparse MCC can be regarded as special cases of the proposed SPF-NMCC algorithm.If γ SPF = 0 and ξ = χ x(n) 2 , the proposed SPF-NMCC algorithm is a soft function parameter penalized MCC algorithm.In addition, we can choose different β to control the SPF-NMCC algorithm.From the derivation of the SPF-NMCC algorithm, we can see that the proposed SPF-NMCC algorithm is a zero attracting algorithm, whose zero attracting ability can be controlled by the regularization parameter γ SPF .In the update Equation (30), −γ SPF S β ( ŵ(n)) is a zero attractor, which is used to speed up the convergence of the SPF-NMCC algorithm.
To better understand the optimization problem given in Equation ( 19), we present an intuitive geometric explanation of the updated equation in Equation (30), which is illustrated in Figure 2. It is found that ŵ(n) converges to the hyperplane II that is defined by ē(n) = 0 through paths A1 and A2.Additionally, path A1 is orthogonal to the hyperplane.Path A2 is the zero-attracting path, which forces the zero or near-zero coefficients of ŵ(n) in the direction to zero.We can also see that the exp − e 2 (n) 2σ 2 approximates to ŵ(n + 1) for large e(n), which helps to reduce the update magnitude to render the proposed algorithm robust in dealing with impulsive noise.

Performance of the SPF-NMCC Algorithm
In this section, we will investigate the estimation performance of the proposed SPF-NMCC algorithm in the context of sparse system identification.The key parameters of the proposed SPF-NMCC algorithm are discussed and the performance of the SPF-NMCC algorithm is evaluated with different sparsity levels.Furthermore, the convergence of the proposed SPF-NMCC algorithm is given and is compared with LMS, MCC, ZA-MCC and RZA-MCC algorithms.In all the simulations, 500 Monte Carlo runs are used to get each point for all the mentioned adaptive filtering algorithms.Herein, the sparsity level K is defined as the number of non-zero coefficients and these K dominant coefficients are randomly distributed within the length of the sparse system.In addition, the FIR system with a length of N = 16 and the sparsity levels of K = 2, K = 4 and K = 8 are considered to investigate the performance of the proposed SPF-NMCC algorithm.Since the MCC algorithms perform well in an impulsive noise environment, we employ an impulsive noise v(n) model, which is given as follows where N(ι i , ν 2 i )(i = 1, 2) are the Gaussian distributions whose means are ι i and variances are ν 2 i , and θ is a mixture parameter used for controlling the mixture of the two noises.In addition, we assume the mixed Gaussian noise is independent of x(n).In all of the simulations, the mixed impulsive noise (ι i , ν 2 i , ι 2 , ν 2 2 , θ) is set to be (0, 0.05, 0, 20, 0.05).The sparse system estimation performance is given by mean square deviation (MSD) whose definition is Firstly, the effects of parameters β, µ SPF , and γ SPF are investigated in detail.The effects of β on the proposed SPF-NMCC algorithm are shown in Figure 3.It is found that the steady-state error floor is reduced with an increment of β because large β can well approximate l 0 -norm.The steady-state behavior of the proposed SPF-NMCC algorithm with various µ SPF is demonstrated in Figure 4.It is noted that the convergence of the proposed SPF-NMCC algorithm becomes faster when µ SPF increases from 0.2 to 0.9, which is similar to the MCC algorithm.Thus, parameters γ SPF , β and µ SPF of the proposed SPF-NMCC algorithm should be properly selected to achieve a good performance.The effects of the regularization parameter γ SPF is investigated to evaluate the steady-state performance of the proposed SPF-NMCC algorithm.In this experiment, the simulation parameters are σ = 1000, β = 6 and ξ = 0.2.The simulated results of the parameter γ SPF are shown in Figure 5.It can be seen that the regularization parameter γ SPF has an important effect on the steady-state performance of the proposed SPF-NMCC algorithm.When γ SPF decreases from 8 × 10 −4 to 5 × 10 −5 , the steady-state error floor is reduced.If we continue to decrease the value of γ SPF , the steady-state error floor is deteriorated and is rebounded to an opposite direction.This is because the large γ SPF can provide a strong zero attraction to attract the zero or near zero coefficients to zero quickly, while the small γ SPF exerts a weak zero attraction on the zero or near zero coefficients [26,28].Thus, the regularization parameter γ SPF can well control the zero attracting ability to provide an excellent performance for sparse system identification.Secondly, we perform an experiment to investigate the convergence speed of the proposed SPF-NMCC algorithm.The convergence of the SPF-NMCC algorithm is illustrated in Figure 6 and its convergence is also compared with the existing LMS, MCC, NLMS, NMCC, ZA-LMS, ZA-MCC, RZA-LMS, and RZA-MCC algorithms.Here, the simulation parameters are µ LMS = 0.005, χ = 0.0052, µ NLMS = 0.082, χ NMCC = 0.087, µ ZALMS = 0.008, ρ ZALMS = 9 × 10 −6 , ξ ZA = 0.008, ρ ZA = 3 × 10 −5 , µ RZALMS = 0.013, ρ RZALMS = 5 × 10 −5 , ξ RZA = 0.018, ρ RZA = 7 × 10 −5 , µ SPF = 0.3, and γ SPF = 2.5 × 10 −5 , which result in nearly the same steady-state error floor.Here, µ LMS , µ NLMS , µ ZALMS and µ RZALMS represent the step-sizes of the LMS, NLMS, ZA-LMS and RZA-LMS algorithms, while ρ ZALMS and ρ RZALMS are the regularization parameters of the ZA-LMS and RZA-LMS algorithms.It is observed that our proposed SPF-NMCC algorithm achieves the fastest convergence speed rate.It is worth noting that the NMCC algorithm also converges faster than the MCC algorithm.Another interesting result of this figure is that zero attracting algorithms increase convergence speed, and further increments on it can be obtained by reweighing the coefficients.It is also worth noting that algorithms using MCC converge faster than those using LMS criterion.Furthermore, the proposed SPF scheme can provide a flexible zero attraction to switch the norm penalties between l 0 -norm and l 1 -norm, resulting in the fastest convergence.
Thirdly, our proposed SPF-NMCC algorithm is analyzed under different sparsity level K, namely, K = 2, K = 4 and K = 8.In this experiment, the related parameters are set as 45, and γ SPF = 8 × 10 −5 .The system estimation behaviors of the SPF-NMCC algorithm for K = 2, K = 4 and K = 8 are demonstrated in Figures 7-9, respectively.It is found that our proposed SPF-NMCC algorithm has the lowest steady-state error floor for K = 2.In addition, the proposed NMCC algorithm with the same initial convergence is better than the MCC algorithm in terms of the steady-state behavior.When K increases from 2 to 8, the steady-state error floor increases.However, our proposed SPF-NMCC algorithm outperforms the previously presented algorithms with respect to the steady-state behavior.From Figures 7-9, the convergence speed of the proposed SPF-NMCC algorithm is a bit larger than that of the other mentioned algorithms, which is to obtain a lower steady-state behavior.As we know, adaptive filtering algorithms have been widely used for echo cancellation.Therefore, we also construct an experiment to discuss the steady-state behavior of the proposed algorithm over an acoustic echo channel.Here, we define the measurement of the sparseness of the used acoustic echo channel as . Figure 10 gives a typical acoustic echo channel whose length is 256, and there are 16 dominant coefficients.In order to better understand the effects of the sparsity, ζ 12 (w o ) = 0.8222 and ζ 12 (w o ) = 0.7362 are employed to evaluate the tracking behavior of the proposed SPF-NMCC algorithm.The parameters are χ = 0.0045, µ NLMS = 1, χ NMCC = 0.9, µ ZALMS = µ RZALMS = 0.005, ρ ZALMS = ρ RZALMS = 2 × 10 −5 , ξ ZA = 0.0035, ρ ZA = ρ RZA = 5 × 10 −6 , ξ RZA = 0.003, µ SPF = 0.9, and γ SPF = 3 × 10 −6 .The simulation results for tracking the acoustic echo channel are shown in Figure 11.It can be seen that our proposed SPF-NMCC algorithm outperforms the conventional LMS and MCC algorithms and their related sparse forms in terms of both the convergence speed rate and steady-state behavior.Even for a reduced sparsity of ζ 12 (w o ) = 0.7362, our proposed SPF-NMCC algorithm still has the best steady-state performance.Thus, we can say that the proposed SPF-NMCC algorithm shows little dependence on the sparsity of the unknown system w o in comparison with the mentioned previously presented adaptive filtering algorithms.Thus, the proposed SPF-NMCC algorithm is more effective for dealing with sparse system identification.At last, we give the complexity of the SPF-NMCC algorithm and compare it with the mentioned algorithms above in Table 1.We use the additions, multiplications, divisions and exponential calculation to give a comparison of the computational complexity.The comparisons of the related computational complexity are given in Table 1.We can see that our SPF-NMCC algorithm has (N + 4K + 1) additions, (N + 5K) multiplications, (N) divisions and (N + 1) exponential calculations, which provides a moderate computational complexity for sparse system identification.

LMS
On the basis of the discussions and analysis aforementioned, we can give a short summary about our proposed SPF-NMCC algorithm for handling sparse system identification.It is found that our proposed SPF-NMCC algorithm has the fastest convergence speed rate for achieving the same steady-state error floor.Moreover, it can also provide the smallest system estimation misalignment in comparison with the LMS and MCC algorithm and their related zero attracting forms.This is because the proposed SPF-NMCC algorithm employs the SPF function to form a desired zero attractor to exploit the sparseness of the systems, and it uses the normalization to give a resistance to the scaling of the input.In addition, the zero attracting abilities of the proposed SPF-NMCC algorithm can be controlled by γ SPF .Our SPF-NMCC algorithm introduced two extra parameters to exploit the sparsity of the traditional MCC algorithm by introducing a zero attractor.Although two extra parameters have been added, which slightly increases the computational complexity, the performance of the SPF-NMCC algorithm is better than the MCC and NMCC algorithms.Even compared with the previously reported ZA-MCC and RZA-MCC algorithms, the SPF-NMCC algorithm has the fastest convergence and lowest estimation error in terms of MSD.However, the proposed SPF-NMCC algorithm has a cost of selecting the proper hyper-parameter values in practical engineering applications because it requires more time and data to select the parameters β and ρ SPF in comparison with the MCC algorithm.Compared with the ZA-MCC and RZA-MCC algorithms, the proposed SPF-NMCC algorithm needs to find a proper β.

Conclusions
A sparse SPF-NMCC algorithm has been proposed for sparse system identification applications.The proposed SPF-NMCC algorithm was realized by using a soft parameter function as a penalty to construct the desired zero attractor in its iterations.Typical sparse systems were used to evaluate the performance of the SPF-NMCC algorithm in terms of both convergence speed and steady-state behaviors.In addition, the key parameters were also investigated to give a clear explanation of the SPF-NMCC algorithm.Simulation results show that the proposed SPF-NMCC algorithm can achieve the fastest convergence speed and smallest steady-state misalignment.However, the proposed algorithm requires more time and data to select the hyper parameters.In the future, we will develop sparse NMCC algorithms with high performance and fewer hyper parameters.

Figure 1 .
Figure 1.Performance of the soft parameter function with varying β.

Figure 2 .
Figure 2. Updating process interpretation for the proposed SPF-NMCC algorithm.