# Robust Regression with Density Power Divergence: Theory, Comparisons, and Data Analysis

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Minimum Density Power Divergence Estimation

## 3. Robust Regression

#### 3.1. M and S Estimation

- It is symmetric and continuously differentiable, and $\rho \left(0\right)=0$;
- there exists a $c>0$ such that $\rho $ is strictly increasing on $[0,c]$ and constant on $[c,\infty )$; and
- it is such that$$K/\rho \left(c\right)=\mathrm{bdp}\phantom{\rule{1.em}{0ex}}\mathrm{with}\phantom{\rule{1.em}{0ex}}0<\mathrm{bdp}\le 0.5,$$

#### 3.2. S Estimation for Power Divergence Regression

#### 3.2.1. The Breakdown Point and the Rho Function

#### 3.2.2. Efficiency, the Psi Function and the Influence Function

## 4. Comparisons of Asymptotic Properties

## 5. Monitoring and Comparisons with Data

**Definition**

**1.**

- The residual sum of squares.
- Changes in the parameter estimates ${\widehat{\beta}}_{i}$ or $\widehat{\sigma}$.
- Measures of correlation between successive sets of residuals, rather than the sum of squares (Riani et al. [20]).

## 6. Discussion

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Appendix A. Rho Functions

**Tukey’s Biweight function**[13]:

**Hampel’s $\rho $ function**[14] (p. 150) has a similar, but less smooth, shape.

**optimal $\rho $ function**.

**Hyperbolic Tangent $\rho $ function**, which, for suitable constants c, k, A, B, and d, is defined as

## References

- Basu, A.; Harris, I.R.; Hjort, N.L.; Jones, M.C. Robust and efficient estimation by minimizing a density power divergence. Biometrika
**1998**, 85, 549–559. [Google Scholar] [CrossRef][Green Version] - Riani, M.; Cerioli, A.; Torti, F. On consistency factors and efficiency of robust S-estimators. TEST
**2014**, 23, 356–387. [Google Scholar] [CrossRef] - Scott, D.W. Parametric Statistical Modeling by Minimum Integrated Square Error. Technometrics
**2001**, 43, 274–285. [Google Scholar] [CrossRef] - Ghosh, A.; Basu, A. Robust estimation for independent non-homogeneous observations using density power divergence with applications to linear regression. Electron. J. Stat.
**2013**, 7, 2420–2456. [Google Scholar] [CrossRef] - Durio, A.; Isaia, E.D. The minimum density power divergence approach in building robust regression models. Informatica (Lithuania)
**2011**, 22, 43–56. [Google Scholar] - Warwick, J.; Jones, M.C. Choosing a robustness tuning parameter. J. Stat. Comput. Simul.
**2005**, 75, 581–588. [Google Scholar] [CrossRef] - Ghosh, A.; Basu, A. Robust estimation for non-homogeneous data and the selection of the optimal tuning parameter: The density power divergence approach. J. Appl. Stat.
**2015**, 42, 2056–2072. [Google Scholar] [CrossRef] - Rousseeuw, P.J.; Yohai, V.J. Robust regression by means of S-estimators. In Robust and Nonlinear Time Series Analysis: Lecture Notes in Statistics 26; Franke, J., Härdle, W., Martin, R.D., Eds.; Springer: New York, NY, USA, 1984; pp. 256–272. [Google Scholar]
- Maronna, R.A.; Martin, R.D.; Yohai, V.J. Robust Statistics: Theory and Methods; Wiley: Chichester, UK, 2006. [Google Scholar]
- Rousseeuw, P.J.; Leroy, A.M. Robust Regression and Outlier Detection; Wiley: New York, NY, USA, 1987. [Google Scholar]
- Basu, A.; Harris, I.R.; Hjort, N.L.; Jones, M.C. Robust and Efficient Estimation by Minimising a Density Power Divergence; Technical Report, 7; Department of Mathematics, University of Oslo: Oslo, Norway, 1997. [Google Scholar]
- Salibian-Barrera, M.; Yohai, V. A fast algorithm for S-regression estimates. J. Comput. Graph. Stat.
**2006**, 15, 414–427. [Google Scholar] [CrossRef] - Beaton, A.E.; Tukey, J.W. The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data. Technometrics
**1974**, 16, 147–185. [Google Scholar] [CrossRef] - Hampel, F.; Ronchetti, E.M.; Rousseeuw, P.; Stahel, W.A. Robust Statistics; Wiley: New York, NY, USA, 1986. [Google Scholar]
- Huber, P.J. Robust Regression: Asymptotics, Conjectures and Monte Carlo. Ann. Stat.
**1973**, 1, 799–821. [Google Scholar] [CrossRef] - Yohai, V.J.; Zamar, R.H. Optimal locally robust M-estimates of regression. J. Stat. Plan. Inference
**1997**, 64, 309–323. [Google Scholar] [CrossRef] - Hössjer, O. On the optimality of S-estimators. Stat. Probabil. Lett.
**1992**, 14, 413–419. [Google Scholar] [CrossRef] - Salini, S.; Cerioli, A.; Laurini, F.; Riani, M. Reliable Robust Regression Diagnostics. Int. Stat. Rev.
**2015**, 84, 99–127. [Google Scholar] [CrossRef] - Jones, M.C.; Hjort, N.L.; Harris, I.R.; Basu, A. A comparison of related density-based minimum divergence estimators. Biometrika
**2001**, 88, 865–873. [Google Scholar] [CrossRef] - Riani, M.; Cerioli, A.; Atkinson, A.C.; Perrotta, D. Monitoring Robust Regression. Electron. J. Stat.
**2014**, 8, 642–673. [Google Scholar] [CrossRef] - Atkinson, A.C.; Riani, M. Robust Diagnostic Regression Analysis; Springer: New York, NY, USA, 2000. [Google Scholar]
- Rousseeuw, P.J. Least median of squares regression. J. Am. Stat. Assoc.
**1984**, 79, 871–880. [Google Scholar] [CrossRef] - Atkinson, A.C.; Riani, M. Distribution theory and simulations for tests of outliers in regression. J. Comput. Graph. Stat.
**2006**, 15, 460–476. [Google Scholar] [CrossRef] - Perrotta, D.; Riani, M.; Torti, F. New robust dynamic plots for regression mixture detection. Adv. Data Anal. Classi.
**2009**, 3, 263–279. [Google Scholar] [CrossRef] - Atkinson, A.C.; Riani, M.; Cerioli, A. The Forward Search: Theory and data analysis (with discussion). J. Korean Stat. Soc.
**2010**, 39, 117–134. [Google Scholar] [CrossRef] - Huber, P.J.; Ronchetti, E.M. Robust Statistics, 2nd ed.; Wiley: New York, NY, USA, 2009. [Google Scholar]
- Hampel, F.; Rousseeuw, P.; Ronchetti, E. The change-of-variance curve and optimal redescending M-estimators. J. Am. Stat. Assoc.
**1985**, 76, 643–648. [Google Scholar] [CrossRef]

**Figure 1.**Dependence of ${\rho}_{\alpha}\left(x\right)$ on $\alpha $, for frequently used values of robustness properties in Table 1. Left-hand panel, three values of breakdown point (bdp); right-hand panel, three values of eff.

**Figure 5.**Breakdown point and efficiency as parameters vary for five rho functions: TB = Tukey biweight; HA = Hampel; OPT = optimal; PD = power divergence and HYP = hyperbolic. The inset is a zoom of the main figure for high breakdown point.

**Figure 6.**Breakdown point and efficiency as parameters vary for the Hampel and hyperbolic rho functions.

**Figure 7.**Regression data: residuals as bdp decreases. Upper panel, Brute Force (BF)-estimation, lower panel S-estimation.

**Figure 8.**Comparison of estimates of $\sigma $ as bdp decreases. Left-hand panel, regression data: right-hand panel, data with moderate outliers.

**Figure 9.**Data with moderate outliers: residuals as bdp decreases. Upper panel, BF-estimation; lower panel S-estimation.

**Table 1.**S power divergence. Values of $\alpha $, bdp, and eff for three frequently used values of each in bold.

$\mathit{\alpha}$ | bdp | $\mathit{eff}$ |
---|---|---|

0 | 0 | 1 |

0.5 | 0.1835 | 0.8381 |

1 | 0.2929 | 0.6495 |

0.7778 | 0.25 | 0.7271 |

1.7778 | 0.4 | 0.4536 |

3 | 0.5 | 0.2894 |

0.4715 | 0.1756 | 0.85 |

0.3522 | 0.14 | 0.9 |

0.2245 | 0.0963 | 0.95 |

0.089 | 0.0417 | 0.99 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Riani, M.; Atkinson, A.C.; Corbellini, A.; Perrotta, D.
Robust Regression with Density Power Divergence: Theory, Comparisons, and Data Analysis. *Entropy* **2020**, *22*, 399.
https://doi.org/10.3390/e22040399

**AMA Style**

Riani M, Atkinson AC, Corbellini A, Perrotta D.
Robust Regression with Density Power Divergence: Theory, Comparisons, and Data Analysis. *Entropy*. 2020; 22(4):399.
https://doi.org/10.3390/e22040399

**Chicago/Turabian Style**

Riani, Marco, Anthony C. Atkinson, Aldo Corbellini, and Domenico Perrotta.
2020. "Robust Regression with Density Power Divergence: Theory, Comparisons, and Data Analysis" *Entropy* 22, no. 4: 399.
https://doi.org/10.3390/e22040399