Minimax Estimation in Regression under Sample Conformity Constraints
Abstract
:1. Introduction
- A description of the uncertainty set in the observation model;
- A class of the admissible estimators;
- An optimality criterion (a loss function) as a function of the argument pair “estimator–uncertain parameter value”.
- is the Borel -algebra of the topological space S (is S is the whole space) or its contraction to the set S (if S is a set of the topological space);
- is a column vector formed by the ordinary or block components ;
- is a row vector formed by the ordinary or block components ;
- is a scalar product of two finite-dimensional vectors;
- is a set of all continuous real-valued functions with the domain ;
- is the Euclidean norm of the vector x;
- is the probability of the event A corresponding to the distribution F;
- is a mathematical expectation of the random vector X with the distribution F;
- is a convex hull of the set .
2. Statement of Problem
2.1. Formulation
- is an unobservable random vector, having an unknown cumulative distribution function (cdf) F;
- is a random unobservable vector with a known cdf dependent on the value of ;
- is a vector of observations;
- is a random vector of observation errors with the known probability density function (pdf) ;
- is a nonrandom function characterizing the observation plant;
- is a nonrandom function characterizing the observation error intensity.
- The outcome space contains all admissible values of the compound vector ;
- -algebra is determined as ;
- The probability measures are determined as:
2.2. Necessary Assumptions Concerning Observation Model
- (i)
- The set is compact.
- (ii)
- Let be a family of all probability distributions with a support lying within the set . The set is itself a convex *–weakly compact [18] subset of .
- (iii)
- The constraint
- (iv)
- The set is nonempty.
- (v)
- .
- (vi)
- pdf for ; ; the function is a regular version of the conditional distribution for .
- (vii)
- The observation noise is uniformly non-degenerate, i.e.,
- (viii)
- The inequalities
- (ix)
- The set of admissible estimators contains only the functions , for which:
2.3. Argumentation
- A result of natural (non-human) impacts;
- A result of some uncontrollable (parasitic) input signals of “the external players”.
3. The Main Result
4. Analysis and Extensions
4.1. Dual Problem: A Numerical Solution
- be a decreasing nonrandom sequence characterizing the approximation accuracy;
- : be a sequence of embedded subdivisions;
4.2. The Least Favorable Distribution in the Light of the Pareto Efficiency
- be a set of the LFDs in the estimation problem (8) without conformity constrains (i.e., as );
- ;
- be an arbitrary fixed conformity level;
- be a solution to the finite-dimensional dual problem;
- be the set of corresponding LFDs.
4.3. Other Conformity Indices
- are available observations;
- is a random vector with unknown distribution F;
- are the observation errors that are i.i.d. centered normalized random values with the pdf .
- (x)
- the constraintThis holds for all and some fixed level . It is called the constraint based on the EDF.
5. Numerical Examples
5.1. Parameter Estimation in the Kalman Observation System
- is an unobservable state trajectory (the autoregression is supposed to be stable);
- are available observations;
- and are vectorizations of independent standard Gaussian discrete-time white noises;
- , c and f are known parameters;
- is an uncertain vector lying in the fixed rectangle .
- The estimate calculated by the moment/substitution method [12]:
- The Bayesian estimate (11) calculated under the assumption that prior distribution of is uniform over the whole uncertainty set ;
- The Bayesian estimate (11) calculated under the assumption that the prior distribution of is uniform over the vertices of ;
- The estimate calculated by the extended Kalman filter (EKF) algorithm [39] and subsequent residual processing;
- The maximum likelihood estimate (MLE) calculated by the expectation/maximization algorithm (EM algorithm) [17].
- Both minimax estimates and converge to the MLE as . Nevertheless, the rate of convergence depends on the specific choice of the loss function ( or in the considered case).
- Both minimax estimates are more conservative than the MLE, because they take into account a chance for other points of the LFD domain to be realized.
- Under an appropriate choice of the confidence ratio r, both minimax estimates become more accurate than other candidates, except for the MLE.
5.2. Parameter Estimation under Additive and Multiplicative Observation Noises
- a is an estimated value;
- is a vector of the i.i.d. unobservable random values (multiplicative noise): ;
- is a vector of the i.i.d. unobservable random values (additive noise): .
- The EDF calculated by the sample ;
- The cdf’s of Y, corresponding to the one-point distribution concentrated at the point q ();
- The cdf , closest to the EDF within the set .
- The minimax estimate under the conformity constraint, based on the EDF, does not converge to the MLE as .
- Under an appropriate choice of the confidence ratio r, the minimax estimate under the EDF constraint becomes more accurate than other candidates, including the MLE.
6. Conclusions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
cdf | cumulative distribution function |
CE | conditional expectation |
EDF | empirical distribution function |
EKF | extended Kalman filter |
EM algorithm | expectation/maximization algorithm |
LFD | least favorable distribution |
MLE | maximum likelihood estimate |
MS-optimal | optimal in the mean square sense |
probability density function | |
QP problem | quadratic programming problem |
Appendix A
Appendix B
Appendix C
Appendix D
References
- Calafiore, G.; El Ghaoui, L. Robust maximum likelihood estimation in the linear model. Automatica 2001, 37, 573–580. [Google Scholar] [CrossRef]
- Kurzhanski, A.B.; Varaiya, P. Dynamics and Control of Trajectory Tubes; Birkhäuser: Basel, Switzerland, 2014. [Google Scholar]
- Matasov, A. Estimators for Uncertain Dynamic Systems; Kluwer: Dortrecht, The Netherlands, 1998. [Google Scholar]
- Borisov, A.V.; Pankov, A.R. Optimal filtering in stochastic discrete-time systems with unknown inputs. IEEE Trans. Autom. Control 1994, 39, 2461–2464. [Google Scholar] [CrossRef]
- Pankov, A.R.; Semenikhin, K.V. Minimax identification of a generalized uncertain stochastic linear model. Autom. Remote Control 1998, 59, 1632–1643. [Google Scholar]
- Poor, V.; Looze, D. Minimax state estimation for linear stochastic systems with noise uncertainty. IEEE Trans. Autom. Control 1981, 26, 902–906. [Google Scholar] [CrossRef]
- Soloviev, V. Towards the Theory of Minimax-Bayesian Estimation. Theory Probab. Its Appl. 2000, 44, 739–754. [Google Scholar] [CrossRef]
- Blackwell, D.; Girshick, M. Theory of Games and Statistical Decisions; Wiley: New York, NY, USA, 1954. [Google Scholar]
- Martin, C.; Mintz, M. Robust filtering and prediction for linear systems with uncertain dynamics: A game-theoretic approach. IEEE Trans. Autom. Control 1983, 28, 888–896. [Google Scholar] [CrossRef]
- Berger, J.O. Statistical Decision Theory and Bayesian Analysis; Springer: Berlin/Heidelberg, Germany, 1985. [Google Scholar]
- Anan’ev, B. Minimax Estimation of Statistically Uncertain Systems Under the Choice of a Feedback Parameter. J. Math. Syst. Estim. Control 1995, 5, 1–17. [Google Scholar]
- Borovkov, A. Mathematical Statistics; Australia Gordon & Breach: Blackburn, Australia, 1998. [Google Scholar]
- Epstein, L.; Ji, S. Ambiguous volatility, possibility and utility in continuous time. J. Math. Econ. 2014, 50, 269–282. [Google Scholar] [CrossRef] [Green Version]
- Borisov, A.V. A posteriori minimax estimation with likelihood constraints. Autom. Remote Control 2012, 73, 1481–1497. [Google Scholar] [CrossRef]
- Arkhipov, A.; Semenikhin, K. Minimax Linear Estimation with the Probability Criterion under Unimodal Noise and Bounded Parameters. Autom. Remote Control 2020, 81, 1176–1191. [Google Scholar] [CrossRef]
- Germeier, Y. Non-Antagonistic Games; Springer: New York, NY, USA, 1986. [Google Scholar]
- Elliott, R.J.; Moore, J.B.; Aggoun, L. Hidden Markov Models: Estimation and Control; Springer: New York, NY, USA, 1995. [Google Scholar]
- Yosida, K. Functional Analysis; Grundlehren der Mathematischen Wissenschaften; Springer: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
- Liptser, R.; Shiryaev, A. Statistics of Random Processes: I. General Theory; Springer: Berlin/Heidelberg, Germany, 2001. [Google Scholar]
- Kats, I.; Kurzhanskii, A. Estimation in Multistep Systems. Proc. USSR Acad. Sci. 1975, 221, 535–538. [Google Scholar]
- Petersen, I.R.; James, M.R.; Dupuis, P. Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Trans. Autom. Control 2000, 45, 398–412. [Google Scholar] [CrossRef] [Green Version]
- Xie, L.; Ugrinovskii, V.A.; Petersen, I.R. Finite horizon robust state estimation for uncertain finite-alphabet hidden Markov models with conditional relative entropy constraints. In Proceedings of the 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601), Nassau, Bahamas, 14–17 December 2004; Volume 4, pp. 4497–4502. [Google Scholar] [CrossRef]
- El Karoui, N.; Jeanblanc Picque, M. Contrôle de processus de Markov. Séminaire Probab. Strasbg. 1988, 22, 508–541. [Google Scholar]
- Lee, E.; Markus, L. Foundations of Optimal Control Theory; SIAM Series in Applied Mathematics; Wiley: Hoboken, NJ, USA, 1967. [Google Scholar]
- Floyd, S.; Jacobson, V. Random early detection gateways for congestion avoidance. IEEE/ACM Trans. Netw. 1993, 1, 397–413. [Google Scholar] [CrossRef]
- Low, S.H.; Paganini, F.; Doyle, J.C. Internet congestion control. IEEE Control Syst. Mag. 2002, 22, 28–43. [Google Scholar] [CrossRef] [Green Version]
- Altman, E.; Avrachenkov, K.; Menache, I.; Miller, G.; Prabhu, B.J.; Shwartz, A. Dynamic Discrete Power Control in Cellular Networks. IEEE Trans. Autom. Control 2009, 54, 2328–2340. [Google Scholar] [CrossRef]
- Perruquetti, W.; Barbot, J.P. Sliding Mode Control in Engineering; Marcel Dekker, Inc.: New York, NY, USA, 2002. [Google Scholar]
- Arnold, B.F.; Stahlecker, P. Fuzzy prior information and minimax estimation in the linear regression model. Stat. Pap. 1997, 38, 377–391. [Google Scholar] [CrossRef]
- Donoho, D.; Johnstone, I.; Stern, A.; Hoch, J. Does the maximum entropy method improve sensitivity? Proc. Natl. Acad. Sci. USA 1990, 87, 5066–5068. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Donoho, D.L.; Johnstone, I.M.; Hoch, J.C.; Stern, A.S. Maximum Entropy and the Nearly Black Object. J. R. Stat. Society. Ser. B 1992, 54, 41–81. [Google Scholar] [CrossRef]
- Pham, D.S.; Bui, H.H.; Venkatesh, S. Bayesian Minimax Estimation of the Normal Model with Incomplete Prior Covariance Matrix Specification. IEEE Trans. Inf. Theory 2010, 56, 6433–6449. [Google Scholar] [CrossRef] [Green Version]
- Donoho, D.L.; Johnstone, I.M. Minimax risk over lp-balls for lq-error. Probab. Theory Relat. Fields 1994, 99, 277–303. [Google Scholar] [CrossRef]
- Donoho, D.L.; Johnstone, I.M. Minimax estimation via wavelet shrinkage. Ann. Stat. 1998, 26, 879–921. [Google Scholar] [CrossRef]
- Donoho, D.L.; Johnstone, I.; Montanari, A. Accurate Prediction of Phase Transitions in Compressed Sensing via a Connection to Minimax Denoising. IEEE Trans. Inf. Theory 2013, 59, 3396–3433. [Google Scholar] [CrossRef] [Green Version]
- Bosov, A.; Borisov, A.; Semenikhin, K. Conditionally Minimax Prediction in Nonlinear Stochastic Systems. IFAC-PapersOnLine 2015, 48, 802–807. [Google Scholar] [CrossRef]
- Kadets, V. A Course in Functional Analysis and Measure Theory; Springer: Berlin/Heidelberg, Germany, 2018. [Google Scholar]
- Gelman, A.; Carlin, J.B.; Stern, H.S.; Rubin, D.B. Bayesian Data Analysis, 2nd ed.; Chapman and Hall/CRC: London, UK, 2004. [Google Scholar]
- Anderson, B.; Moore, J. Optimal Filtering; Prentice-Hall: Upper Saddle River, NJ, USA, 1979. [Google Scholar]
- Grabiner, J.; Balakrishnan, A. Applications of Mathematics: Applied Functional Analysis; Applications of Mathematics; Springer: New York, NY, USA, 1981. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Borisov, A. Minimax Estimation in Regression under Sample Conformity Constraints. Mathematics 2021, 9, 1080. https://doi.org/10.3390/math9101080
Borisov A. Minimax Estimation in Regression under Sample Conformity Constraints. Mathematics. 2021; 9(10):1080. https://doi.org/10.3390/math9101080
Chicago/Turabian StyleBorisov, Andrey. 2021. "Minimax Estimation in Regression under Sample Conformity Constraints" Mathematics 9, no. 10: 1080. https://doi.org/10.3390/math9101080
APA StyleBorisov, A. (2021). Minimax Estimation in Regression under Sample Conformity Constraints. Mathematics, 9(10), 1080. https://doi.org/10.3390/math9101080