MDPI - Publisher of Open Access Journals

21 pages, 4425 KiB

Open AccessArticle

The Prediction Performance Analysis of the Lasso Model with Convex Non-Convex Sparse Regularization

by Wei Chen, Qiuyue Liu, Hancong Li and Jian Zou

Algorithms 2025, 18(4), 195; https://doi.org/10.3390/a18040195 - 1 Apr 2025

Viewed by 457

The incorporation of

ℓ_{1}

regularization in Lasso regression plays a crucial role by inducing convexity to the objective function, thereby facilitating its minimization; when compared to non-convex regularization, the utilization of

ℓ_{1}

regularization introduces bias through artificial coefficient shrinkage towards zero. [...] Read more.

The incorporation of

ℓ_{1}

regularization in Lasso regression plays a crucial role by inducing convexity to the objective function, thereby facilitating its minimization; when compared to non-convex regularization, the utilization of

ℓ_{1}

regularization introduces bias through artificial coefficient shrinkage towards zero. Recently, the convex non-convex (CNC) regularization framework has emerged as a powerful technique that enables the incorporation of non-convex regularization terms while maintaining the overall convexity of the optimization problem. Although this method has shown remarkable performance in various empirical studies, its theoretical understanding is still relatively limited. In this paper, we provide a theoretical investigation into the prediction performance of the Lasso model with CNC sparse regularization. By leveraging oracle inequalities, we establish a tighter upper bound on prediction performance compared to the traditional

ℓ_{1}

regularizer. Additionally, we propose an alternating direction method of multipliers (ADMM) algorithm to efficiently solve the proposed model and rigorously analyze its convergence property. Our numerical results, evaluated on both synthetic data and real-world magnetic resonance imaging (MRI) reconstruction tasks, confirm the superior effectiveness of our proposed approach. Full article

(This article belongs to the Section Analysis of Algorithms and Complexity Theory)

► Show Figures

Figure 1

16 pages, 366 KiB

Open AccessArticle

A Method for Transforming Non-Convex Optimization Problem to Distributed Form

by Oleg O. Khamisov, Oleg V. Khamisov, Todor D. Ganchev and Eugene S. Semenkin

Mathematics 2024, 12(17), 2796; https://doi.org/10.3390/math12172796 - 9 Sep 2024

Cited by 3 | Viewed by 1740

Abstract

We propose a novel distributed method for non-convex optimization problems with coupling equality and inequality constraints. This method transforms the optimization problem into a specific form to allow distributed implementation of modified gradient descent and Newton’s methods so that they operate as if [...] Read more.

We propose a novel distributed method for non-convex optimization problems with coupling equality and inequality constraints. This method transforms the optimization problem into a specific form to allow distributed implementation of modified gradient descent and Newton’s methods so that they operate as if they were distributed. We demonstrate that for the proposed distributed method: (i) communications are significantly less time-consuming than oracle calls, (ii) its convergence rate is equivalent to the convergence of Newton’s method concerning oracle calls, and (iii) for the cases when oracle calls are more expensive than communication between agents, the transition from a centralized to a distributed paradigm does not significantly affect computational time. The proposed method is applicable when the objective function is twice differentiable and constraints are differentiable, which holds for a wide range of machine learning methods and optimization setups. Full article

(This article belongs to the Special Issue Applied and Computational Mathematics for Digital Environments, 2nd Edition)

► Show Figures

Figure 1

21 pages, 1457 KiB

Open AccessArticle

Variable Selection for Sparse Logistic Regression with Grouped Variables

by Mingrui Zhong, Zanhua Yin and Zhichao Wang

Mathematics 2023, 11(24), 4979; https://doi.org/10.3390/math11244979 - 17 Dec 2023

Viewed by 1793

Abstract

We present a new penalized method for estimation in sparse logistic regression models with a group structure. Group sparsity implies that we should consider the Group Lasso penalty. In contrast to penalized log-likelihood estimation, our method can be viewed as a penalized weighted [...] Read more.

We present a new penalized method for estimation in sparse logistic regression models with a group structure. Group sparsity implies that we should consider the Group Lasso penalty. In contrast to penalized log-likelihood estimation, our method can be viewed as a penalized weighted score function method. Under some mild conditions, we provide non-asymptotic oracle inequalities promoting the group sparsity of predictors. A modified block coordinate descent algorithm based on a weighted score function is also employed. The net advantage of our algorithm over existing Group Lasso-type procedures is that the tuning parameter can be pre-specified. The simulations show that this algorithm is considerably faster and more stable than competing methods. Finally, we illustrate our methodology with two real data sets. Full article

(This article belongs to the Section D1: Probability and Statistics)

► Show Figures

Figure 1

16 pages, 315 KiB

Open AccessArticle

On a Low-Rank Matrix Single-Index Model

by The Tien Mai

Mathematics 2023, 11(9), 2065; https://doi.org/10.3390/math11092065 - 26 Apr 2023

Cited by 2 | Viewed by 1507

Abstract

In this paper, we conduct a theoretical examination of a low-rank matrix single-index model. This model has recently been introduced in the field of biostatistics, but its theoretical properties for jointly estimating the link function and the coefficient matrix have not yet been [...] Read more.

In this paper, we conduct a theoretical examination of a low-rank matrix single-index model. This model has recently been introduced in the field of biostatistics, but its theoretical properties for jointly estimating the link function and the coefficient matrix have not yet been fully explored. In this paper, we make use of the PAC-Bayesian bounds technique to provide a thorough theoretical understanding of the joint estimation of the link function and the coefficient matrix. This allows us to gain a deeper insight into the properties of this model and its potential applications in different fields. Full article

(This article belongs to the Special Issue New Advances in High-Dimensional and Non-asymptotic Statistics)

18 pages, 689 KiB

Open AccessArticle

Selectivity Estimation of Inequality Joins in Databases

by Diogo Repas, Zhicheng Luo, Maxime Schoemans and Mahmoud Sakr

Mathematics 2023, 11(6), 1383; https://doi.org/10.3390/math11061383 - 13 Mar 2023

Cited by 1 | Viewed by 3096

Abstract

Selectivity estimation refers to the ability of the SQL query optimizer to estimate the size of the results of a predicate in the query. It is the main calculation based on which the optimizer can select the least expensive plan to execute. While [...] Read more.

Selectivity estimation refers to the ability of the SQL query optimizer to estimate the size of the results of a predicate in the query. It is the main calculation based on which the optimizer can select the least expensive plan to execute. While the problem has been known since the mid-1970s, we were surprised that there are no solutions in the literature for the selectivity estimation of inequality joins. By testing four common database systems: Oracle, SQL-Server, PostgreSQL, and MySQL, we found that the open-source systems PostgreSQL and MySQL lack this estimation. Oracle and SQL-Server make fairly accurate estimations, yet their algorithms are secret. This paper, thus, proposes an algorithm for inequality join selectivity estimation. The proposed algorithm was implemented in PostgreSQL and sent as a patch to be included in the next releases. We compared this implementation with the above DBMS for three different data distributions (uniform, normal, and Zipfian) and showed that our algorithm provides extremely accurate estimations (below 0.1% average error), outperforming the other systems by an order of magnitude. Full article

(This article belongs to the Special Issue Numerical Methods for Approximation of Functions and Data)

► Show Figures

Figure 1

15 pages, 409 KiB

Open AccessArticle

Group Logistic Regression Models with l_p,q Regularization

by Yanfang Zhang, Chuanhua Wei and Xiaolin Liu

Mathematics 2022, 10(13), 2227; https://doi.org/10.3390/math10132227 - 25 Jun 2022

Cited by 9 | Viewed by 2279

Abstract

In this paper, we proposed a logistic regression model with

l_{p, q}

regularization that could give a group sparse solution. The model could be applied to variable-selection problems with sparse group structures. In the context of big data, the solutions for [...] Read more.

In this paper, we proposed a logistic regression model with

l_{p, q}

regularization that could give a group sparse solution. The model could be applied to variable-selection problems with sparse group structures. In the context of big data, the solutions for practical problems are often group sparse, so it is necessary to study this kind of model. We defined the model from three perspectives: theoretical, algorithmic and numeric. From the theoretical perspective, by introducing the notion of the group restricted eigenvalue condition, we gave the oracle inequality, which was an important property for the variable-selection problems. The global recovery bound was also established for the logistic regression model with

l_{p, q}

regularization. From the algorithmic perspective, we applied the well-known alternating direction method of multipliers (ADMM) algorithm to solve the model. The subproblems for the ADMM algorithm were solved effectively. From the numerical perspective, we performed experiments for simulated data and real data in the factor stock selection. We employed the ADMM algorithm that we presented in the paper to solve the model. The numerical results were also presented. We found that the model was effective in terms of variable selection and prediction. Full article

(This article belongs to the Special Issue New Advances in High-Dimensional and Non-asymptotic Statistics)

► Show Figures

Figure 1

25 pages, 414 KiB

Open AccessEditor’s ChoiceArticle

Heterogeneous Overdispersed Count Data Regressions via Double-Penalized Estimations

by Shaomin Li, Haoyu Wei and Xiaoyu Lei

Mathematics 2022, 10(10), 1700; https://doi.org/10.3390/math10101700 - 16 May 2022

Cited by 5 | Viewed by 2741

Abstract

Recently, the high-dimensional negative binomial regression (NBR) for count data has been widely used in many scientific fields. However, most studies assumed the dispersion parameter as a constant, which may not be satisfied in practice. This paper studies the variable selection and dispersion [...] Read more.

Recently, the high-dimensional negative binomial regression (NBR) for count data has been widely used in many scientific fields. However, most studies assumed the dispersion parameter as a constant, which may not be satisfied in practice. This paper studies the variable selection and dispersion estimation for the heterogeneous NBR models, which model the dispersion parameter as a function. Specifically, we proposed a double regression and applied a double

ℓ_{1}

-penalty to both regressions. Under the restricted eigenvalue conditions, we prove the oracle inequalities for the lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, derived from the oracle inequalities, the consistency and convergence rate for the estimators are the theoretical guarantees for further statistical inference. Finally, both simulations and a real data analysis demonstrate that the new methods are effective. Full article

(This article belongs to the Special Issue New Advances in High-Dimensional and Non-asymptotic Statistics)

28 pages, 4149 KiB

Open AccessArticle

Sparse Density Estimation with Measurement Errors

by Xiaowei Yang, Huiming Zhang, Haoyu Wei and Shouzheng Zhang

Entropy 2022, 24(1), 30; https://doi.org/10.3390/e24010030 - 24 Dec 2021

Viewed by 2872

Abstract

This paper aims to estimate an unknown density of the data with measurement errors as a linear combination of functions from a dictionary. The main novelty is the proposal and investigation of the corrected sparse density estimator (CSDE). Inspired by the penalization approach, [...] Read more.

This paper aims to estimate an unknown density of the data with measurement errors as a linear combination of functions from a dictionary. The main novelty is the proposal and investigation of the corrected sparse density estimator (CSDE). Inspired by the penalization approach, we propose the weighted Elastic-net penalized minimal

ℓ_{2}

-distance method for sparse coefficients estimation, where the adaptive weights come from sharp concentration inequalities. The first-order conditions holding a high probability obtain the optimal weighted tuning parameters. Under local coherence or minimal eigenvalue assumptions, non-asymptotic oracle inequalities are derived. These theoretical results are transposed to obtain the support recovery with a high probability. Some numerical experiments for discrete and continuous distributions confirm the significant improvement obtained by our procedure when compared with other conventional approaches. Finally, the application is performed in a meteorology dataset. It shows that our method has potency and superiority in detecting multi-mode density shapes compared with other conventional approaches. Full article

(This article belongs to the Topic Machine and Deep Learning)

► Show Figures

Figure 1

35 pages, 507 KiB

Open AccessArticle

A Refutation of Finite-State Language Models through Zipf’s Law for Factual Knowledge

by Łukasz Dębowski

Entropy 2021, 23(9), 1148; https://doi.org/10.3390/e23091148 - 1 Sep 2021

Cited by 4 | Viewed by 3668

Abstract

We present a hypothetical argument against finite-state processes in statistical language modeling that is based on semantics rather than syntax. In this theoretical model, we suppose that the semantic properties of texts in a natural language could be approximately captured by a recently [...] Read more.

We present a hypothetical argument against finite-state processes in statistical language modeling that is based on semantics rather than syntax. In this theoretical model, we suppose that the semantic properties of texts in a natural language could be approximately captured by a recently introduced concept of a perigraphic process. Perigraphic processes are a class of stochastic processes that satisfy a Zipf-law accumulation of a subset of factual knowledge, which is time-independent, compressed, and effectively inferrable from the process. We show that the classes of finite-state processes and of perigraphic processes are disjoint, and we present a new simple example of perigraphic processes over a finite alphabet called Oracle processes. The disjointness result makes use of the Hilberg condition, i.e., the almost sure power-law growth of algorithmic mutual information. Using a strongly consistent estimator of the number of hidden states, we show that finite-state processes do not satisfy the Hilberg condition whereas Oracle processes satisfy the Hilberg condition via the data-processing inequality. We discuss the relevance of these mathematical results for theoretical and computational linguistics. Full article

(This article belongs to the Special Issue Information-Theoretic Approaches to Explaining Linguistic Structure)

16 pages, 446 KiB

Open AccessArticle

Random Sampling Many-Dimensional Sets Arising in Control

by Pavel Shcherbakov, Mingyue Ding and Ming Yuchi

Mathematics 2021, 9(5), 580; https://doi.org/10.3390/math9050580 - 9 Mar 2021

Viewed by 2273

Abstract

Various Monte Carlo techniques for random point generation over sets of interest are widely used in many areas of computational mathematics, optimization, data processing, etc. Whereas for regularly shaped sets such sampling is immediate to arrange, for nontrivial, implicitly specified domains these techniques [...] Read more.

Various Monte Carlo techniques for random point generation over sets of interest are widely used in many areas of computational mathematics, optimization, data processing, etc. Whereas for regularly shaped sets such sampling is immediate to arrange, for nontrivial, implicitly specified domains these techniques are not easy to implement. We consider the so-called Hit-and-Run algorithm, a representative of the class of Markov chain Monte Carlo methods, which became popular in recent years. To perform random sampling over a set, this method requires only the knowledge of the intersection of a line through a point inside the set with the boundary of this set. This component of the Hit-and-Run procedure, known as boundary oracle, has to be performed quickly when applied to economy point representation of many-dimensional sets within the randomized approach to data mining, image reconstruction, control, optimization, etc. In this paper, we consider several vector and matrix sets typically encountered in control and specified by linear matrix inequalities. Closed-form solutions are proposed for finding the respective points of intersection, leading to efficient boundary oracles; they are generalized to robust formulations where the system matrices contain norm-bounded uncertainty. Full article

(This article belongs to the Special Issue Machine Learning and Data Mining in Pattern Recognition)

► Show Figures

Figure 1

11 pages, 793 KiB

Open AccessArticle

Adaptive Wavelet Estimations in the Convolution Structure Density Model

by Kaikai Cao and Xiaochen Zeng

Mathematics 2020, 8(9), 1391; https://doi.org/10.3390/math8091391 - 19 Aug 2020

Cited by 2 | Viewed by 1608

Abstract

Using kernel methods, Lepski and Willer study a convolution structure density model and establish adaptive and optimal

L^{p}

risk estimations over an anisotropic Nikol’skii space (Lepski, O.; Willer, T. Oracle inequalities and adaptive estimation in the convolution structure density model. Ann. Stat. [...] Read more.

Using kernel methods, Lepski and Willer study a convolution structure density model and establish adaptive and optimal

L^{p}

risk estimations over an anisotropic Nikol’skii space (Lepski, O.; Willer, T. Oracle inequalities and adaptive estimation in the convolution structure density model. Ann. Stat.2019, 47, 233–287). Motivated by their work, we consider the same problem over Besov balls by wavelets in this paper and first provide a linear wavelet estimate. Subsequently, a non-linear wavelet estimator is introduced for adaptivity, which attains nearly-optimal convergence rates in some cases. Full article

Search Results (11)

Further Information

Guidelines

MDPI Initiatives

Follow MDPI

Saved Queries

Search Filter Reset All

Years

Feature Papers

Subjects

Journals

Article Types

Countries / Regions

Search Results (11)

Further Information

Guidelines

MDPI Initiatives

Follow MDPI