An Entropic Approach to Constrained Linear Regression

Abstract

We introduce a novel entropy minimization approach for the solution of constrained linear regression problems. Rather than minimizing the quadratic error, our method minimizes the Fermi–Dirac entropy, with the problem data incorporated as constraints. In addition to providing a solution to the linear regression problem, this approach also estimates the measurement error. The only prior assumption made about the errors is analogous to the assumption made about the unknown regression coefficients: specifically, the size of the interval within which they are expected to lie. We compare the results of our approach with those obtained using the disciplined convex optimization methodology. Furthermore, we address consistency issues and present examples to illustrate the effectiveness of our method.

1. Introduction

In the statistical analysis of input–response systems, many prediction or modeling problems involve establishing a functional relationship between input data and output (response) measurements. The input data can be viewed as the control variables of the experiments or as appropriately chosen functions derived from the control data. These inputs are organized into a design matrix $\mathit{X}$, where each row represents the components of the (transposed) input vector. In this context, the responses to the inputs are treated as real numbers. The linear regression problem consists of solving the following problem.

Find vector $\mathit{\alpha }\in {\mathcal{K}}_a\subset {\mathbb{R}}^{K-1}$ and scalar $\beta \in {\mathcal{K}}_b\subset \mathbb{R}$ such that

$$\begin{array}{cc}\hfill & \mathrm{Solve}\mathit{y}=\mathit{X}\mathit{\alpha }+\beta {\mathbf{1}}_N,\hfill \\ \hfill \mathrm{subject} \mathrm{to}\mathit{\alpha }\in {\mathcal{K}}_a& =\prod _{j=1}^{K-1}[l_j,L_j],\beta \in {\mathcal{K}}_b=[h,H]\hfill \end{array}$$

(1)

Here, we temporarily use the standard notations in the statistical literature. Below, we use generic algebraic notations to solve the inverse problem. In this problem, the vector $\mathit{\alpha }$ explains how the different inputs are coupled to obtain the outputs. Thus, the data of the problem consist of the observed outputs, modeled by the vector $\mathit{y}\in {\mathbb{R}}^N$ and the $N\times (K-1)$ design matrix $\mathit{X}$. The symbol ${\mathbf{1}}_N$ stands for the N-vector with all components equal to 1 and, as noted, $\beta \in \mathbb{R}$ is a scalar.

The intuitive meaning of the constraint upon ${\alpha }_j$ amounts to prior knowledge of a range for $\partial \mathit{y}/\partial X_j$, where $X_j$ is the j-th column of $\mathit{X},$ i.e., of the sensitivity of the response to a change in the input (or control) variable. The specification of ${\mathcal{K}}_b$ amounts to the specification of a range for the intercept of the regression. These values are either obtained from an underlying model or are to be inferred from the data, as shown in the first example in Section 3.

Let us begin by rewriting the problem as follows. We think of vectors as column vectors, and, for any vector $\mathit{v}$ or matrix $\mathit{A},$ we shall denote by ${\mathit{v}}^t$ or ${\mathit{A}}^t$, respectively, its transpose; $⟨\mathit{v},\mathit{w}⟩={\mathit{v}}^t\mathit{w}$ denotes the usual Euclidean scalar product of the vectors $\mathit{v}$ and $\mathit{w}$; $\parallel \mathit{v}\parallel$ is the Euclidean norm of $\mathit{v}$; and $[··]$ will denote the concatenation of the indicated objects, so, for example, $\left[\mathit{A}\mathit{v}\right]$ concatenates $v$ as an extra column to matrix $\mathit{A}.$ Let us introduce

$${\mathit{A}}_0=\left[\mathit{X}{\mathbf{1}}_N\right],\mathit{x}={\left[{\mathit{\alpha }}^t\beta \right]}^t$$

(2)

Note that ${\mathit{A}}_0$ is now a given $N\times K$ matrix. Then, problem (1) can be restated as

$$\begin{array}{c}\mathrm{Solve} \mathit{y}={\mathit{A}}_0\mathit{x},\\ \mathrm{subject} \mathrm{to}\mathit{x}\in {\mathcal{K}}_a\times {\mathcal{K}}_b\subset {\mathbb{R}}^K.\end{array}$$

(3)

This is a standard constrained inverse problem. The two standard methods to solve it are as follows. Since the problem may have infinitely many solutions (the kernel of ${\mathit{A}}_0$ may be non-0), one way to choose among them is to solve the following variational problem:

$$\begin{array}{cc}\hfill \mathrm{Find}& {\mathit{x}}^{*}={\min \{\parallel \mathit{x}\parallel }^2:\mathit{x}\in {\mathcal{K}}_a\times {\mathcal{K}}_b\subset {\mathbb{R}}^K\},\hfill \\ \hfill \mathrm{subject} \mathrm{to}& \mathit{y}={\mathit{A}}_0\mathit{x}\mathrm{given}\mathit{y}\in {\mathbb{R}}^N.\hfill \end{array}$$

(4)

Here, $\parallel \mathit{x}\parallel$ denotes the standard Euclidean norm in ${\mathbb{R}}^K.$ A related approach motivated by the fact that the output $\mathit{y}$ may not be in the range of ${\mathit{A}}_0$ consists of solving

$$\begin{array}{cc}\hfill \mathrm{Find}& {\mathit{x}}^{*}=\min \{d({\mathit{A}}_0\mathit{x},\mathit{y}):\mathit{x}\in {\mathcal{K}}_a\times {\mathcal{K}}_b\subset {\mathbb{R}}^K\},\hfill \\ \hfill \mathrm{subject} \mathrm{to}& \mathit{y}={\mathit{A}}_0\mathit{x}\mathrm{given}\mathit{y}\in {\mathbb{R}}^N.\hfill \end{array}$$

(5)

Here, $d({\mathit{y}}_1,{\mathit{y}}_2)$ stands for a distance function on ${\mathbb{R}}^N.$ Usually, it is the distance derived from the standard Euclidean norm. However, we stress the fact that these may be taken as arbitrary norms. For such approaches in inverse problems and applications, see [1,2], and, for statistics and econometrics, see [3,4].

The consistency condition for problems (3) and (4) is that $\mathit{y}\in \mathit{int}\left(\left\{{\mathit{A}}_0\mathit{x}\right|\mathit{x}\in {\mathcal{K}}_a\times {\mathcal{K}}_b\}\right)$ (the interior of the range of the constraint set), which may not be the case due to measurement errors. This leads to the statement of the problem as (5). Instead of this, we propose an extension of (1) or equivalently of (3), which has interesting interpretations. The extended version is stated as

$$\begin{array}{cc}\hfill \mathrm{Solve}& \mathit{y}=\mathit{X}\alpha +\beta {\mathbf{1}}_N+\mathit{ϵ},\hfill \\ \hfill \mathrm{subject} \mathrm{to}& \mathit{\alpha }\in {\mathcal{K}}_a=\prod _{j=1}^{K-1}[l_j,L_j],\beta \in {\mathcal{K}}_b=[h,H],\hfill \\ \hfill & \mathit{ϵ}\in {\mathcal{K}}_e={[-d,d]}^N.\hfill \end{array}$$

(6)

Two interpretations can be drawn. From the perspective of mathematical programming, $\mathit{ϵ}$ can be viewed as a slack variable that absorbs the discrepancy between the system’s response, represented by ${\mathit{A}}_0$, and the observed signal $\mathit{y}.$ Put differently, $\mathit{y}$ now lies within the range of the extended operator $\mathit{A}$, which acts on the augmented domain ${\mathcal{K}}_a\times {\mathcal{K}}_b\times {\mathcal{K}}_e.$

From a statistical standpoint, Equation (6) can be interpreted as simultaneously determining both the regression coefficients and the measurement noise. This is particularly relevant because it can help to uncover systematic errors in the measurement process—an important consideration when data are scarce or expensive to obtain. This approach is worth examining alongside the Bayesian framework for linear regression, as described in [5,6], which include numerous applications to classification and machine learning. However, unlike the standard Bayesian approach, which is tied to the methodology in (5) and relies on a parametric (Gaussian) noise model, our proposal is model-free and non-parametric. In contrast, the noise vector estimated through our method can serve as a starting point for the construction of a model of the noise present in the measurements.

To simplify the notations for the mathematical procedure, consider the $N\times (K+N)$ matrix $\mathit{A}=\left[{\mathit{A}}_0{\mathbb{I}}_N\right],$ where ${\mathbb{I}}_N$ stands for the N-dimensional identity matrix, $\mathit{\xi }={\left[{\mathit{x}}^t{\mathit{ϵ}}^t\right]}^t$ for the unknown of the problem, and $\mathcal{K}={\mathcal{K}}_a\times {\mathcal{K}}_b\times {\mathcal{K}}_e$ for the set of constraints. With this, we restate (6) as the problem

$$\begin{array}{c}\hfill \mathrm{Solve}\mathit{y}=\mathit{A}\mathit{\xi },\\ \hfill \mathrm{subject} \mathrm{to}\mathit{\xi }\in \mathcal{K}\end{array}$$

(7)

Observe that the dimension of the unknown vector is $K+N$, whereas that of the data vector is $N,$ so we have a truly ill-posed problem to solve. Two standard approaches to solving this class of problems were already outlined in (4) and (5). From the abstract point of view, they consist of minimizing a convex function subject to linear and domain constraints, and standard quadratic optimization methods, like discipline convex optimization or CVX [7], can be used.

Our proposal is similar in essence but different in detail. We propose to minimize a convex function (the Fermi–Dirac entropy) defined on the constraint set, subject to a linear constraint, that automatically yields an optimizer in the interior of the constraint set. It is in this detail that the difference from quadratic minimization lies.

We also mention that the combination of linear regression and the minimization of the quadratic distance is used in the context of reproducing kernel Hilbert spaces for learning and classification problems, as in the works [8,9]. Besides the standard applications in statistical analysis, and the newer applications in statistical learning already mentioned, we mention the application of linear regression in decoding, as considered by [10].

It is worth noting that our approach provides an alternative to the maximum entropy in the mean methods proposed in [11], as well as its measure-theoretic formulation in [12]. The key difference between our approach and these methods lies in their foundational principles. The maximum entropy in the mean method involves transforming the algebraic problem into one of determining a probability distribution over the set of constraints. In this framework, the solution to the algebraic problem corresponds to the expected value of a random variable with respect to an unknown probability distribution, which is determined using the standard maximum entropy method. In contrast, our approach starts with a Fermi–Dirac-type entropy defined directly on the set of constraints. By minimizing this entropy, we obtain a solution that lies directly within the constraint set.

For optimization purposes, it is useful to note that the entropy function is the Lagrange–Fenchel dual of the logarithm of the Laplace transform of a measure whose support is the constraint space.

The remainder of this paper is organized as follows. In the next section, we collect the mathematical details of the method. Some geometric aspects of the related inverse problem are examined in [13]. At the end of Section 2, we explicitly explain how we measure the quality of the solution. Although the measure is similar to that used in the quadratic optimization procedure, its origin is quite different. The main difference is that our proposal yields a solution (when it exists) in the interior of the constraint set. In Section 3, we consider several variations of the theme of two toy examples: a textbook example and a simulated example. The first of them is an example with a small number of data points, and we use them to exemplify how to determine the constraint set. We also examine the performance of the method for different sizes of a simulated design matrix and of a data vector, and we compare them to the solutions to the same problems obtained applying the CVX method. In Section 4, we address a consistency issue: we verify that our method is consistent with the obvious algebraic solution when the matrix $\mathit{A}$ is invertible or when it is of full rank with ${\mathit{A}}^t\mathit{A}$ invertible. We conclude with some remarks.

2. The Entropy Minimization Approach

Clearly, problem (7) is an ill-posed linear inverse problem with convex constraints. Instead of using the traditional least squares methodology, our approach consists of devising a smooth convex function $\mathrm{\Psi }\left(\mathit{\xi }\right):\mathcal{K}\to \mathbb{R}$ and solving

$$\mathrm{Find}{\mathit{\xi }}^{*}=argmin\{\mathrm{\Psi }\left(\mathit{\xi }\right):\mathit{\xi }\in \mathcal{K},\mathrm{and}\mathit{A}\mathit{\xi }=\mathit{y}\}.$$

(8)

To begin with, to avoid excessive notations, we write $\mathcal{K}={\mathcal{K}}_a\times {\mathcal{K}}_b\times {\mathcal{K}}_e={\prod }_{j=1}^M[a_j,b_j]$ with $M=K+N.$ The correspondence with the prior notation is clear: the first $K-1$ labels correspond to ${\mathcal{K}}_a$, the K-th to ${\mathcal{K}}_b$, and the last N to ${\mathcal{K}}_e$. On the Borel sets of $\mathcal{K}$, we define a measure $dQ\left(\mathit{\xi }\right)$ that charges its “corners” as follows:

$$dQ\left(\mathit{\xi }\right)={\otimes }_{j=1}^M\left({\delta }_{a_j}\left(d{\xi }_j\right)+{\delta }_{b_j}\left(d{\xi }_j\right)\right).$$

(9)

We use ${\delta }_a\left(dx\right)$ to denote the unit point mass at the point a (that is, the Dirac delta measure at a). There are three reasons for this proposal. First, the convex hull of the support of ${\delta }_a\left(dx\right)+{\delta }_b\left(dx\right)$ is the interval $[a,b]$. Second, the function $\zeta \left(\tau \right)=e^{a\tau }+e^{b\tau }$ introduced in Equation (10) is defined for all $\tau \in \mathbb{R}$. Third, below, we need the invertibility of the mapping $\mathit{\tau }\to {\nabla }_{\tau }M\left(\mathit{\tau }\right).$ As the coordinates are separated, this follows from the fact that the equation $\eta =d\zeta \left(\tau \right)/d\tau$ has a solution if and only if $\eta \in (a,b).$ Having mentioned these preliminaries, the Laplace transform of $dQ$ is easy to compute:

$$\mathit{\tau }\in {\mathbb{R}}^M\to \zeta \left(\mathit{\tau }\right)={\int }_{\mathcal{K}}{e}^{⟨\mathit{\tau },\mathit{\xi }⟩}dQ\left(\mathit{\xi }\right)=\prod _{j=1}^M\left(e^{a_j{\tau }_j}+e^{b_j{\tau }_j}\right)$$

(10)

Moreover, the moment generating function is defined by

$$M\left(\tau \right)=\ln \zeta \left(\mathit{\tau }\right)=\sum _{j=1}^M\ln \left(e^{a_j{\tau }_j}+e^{b_j{\tau }_j}\right).$$

(11)

Then, the function $\mathrm{\Psi }\left(\mathit{\xi }\right)$ that we seek is defined to be the Lagrange-Fenchel dual of $M\left(\mathit{\tau }\right),$ which is given as

$$\mathrm{\Psi }\left(\mathit{\xi }\right)=\underset{\mathit{\tau }}{\sup }\left(⟨\mathit{\xi },\mathit{\tau }⟩-M\left(\mathit{\tau }\right)\right).$$

(12)

Making use of the preliminaries mentioned above, a calculation shows that

$$\mathrm{\Psi }\left(\mathit{\xi }\right)=\sum _{j=1}^M{\displaystyle \frac{{\xi }_j-a_j}{b_j-a_j}}\ln \left({\displaystyle \frac{{\xi }_j-a_j}{b_j-a_j}}\right)+{\displaystyle \frac{b_j-{\xi }_j}{b_j-a_j}}\ln \left({\displaystyle \frac{b_j-{\xi }_j}{b_j-a_j}}\right)$$

(13)

A good reference for these matters is [14]. It is known and standard to verify that $M\left(\mathit{\tau }\right)$ is strongly convex and infinitely differentiable in ${\mathbb{R}}^M$ and $\mathrm{\Psi }\left(\mathit{\xi }\right)$ is strongly convex and infinitely differentiable in $int\left(\mathcal{K}\right)$ and reaches its minimum value in $int\left(\mathcal{K}\right).$ Moreover, for $\mathit{\eta }$ in the interior of $\mathcal{K},$ the equation $\mathit{\eta }=\nabla \mathrm{\Psi }\left(\mathit{\xi }\right)$ has a unique solution, which is easy to determine analytically. Furthermore, the Lagrange–Fenchel dual of $\mathrm{\Psi }\left(\mathit{\xi }\right)$ is $M\left(\mathit{\tau }\right)$, and

$${\nabla }_{\mathit{\tau }}M\left(\mathit{\tau }\right)={\left({\nabla }_{\mathit{\xi }}\mathrm{\Psi }\right)}^{-1}\left(\mathit{\tau }\right),\mathrm{when}{\nabla }_{\mathit{\xi }}\mathrm{\Psi }\left(\mathit{\xi }\right)=\mathit{\tau }.$$

(14)

Having made explicit the objects that we need, the solution to problem (8) can be obtained as stated in the following theorem.

Theorem 1.

Let $\mathrm{\Psi }\left(\mathit{\xi }\right)$ be as in (13) and let $M\left(\mathit{\tau }\right)$ be its Lagrange–Fenchel dual. Suppose that $\mathit{y}\in int\left(\mathit{A}\right(\mathcal{K}\left)\right)$. Then, the solution to (8) is given by

$${\xi }_j^{*}={\displaystyle \frac{a_j{e}^{a_j{\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)}_j}+b_j{e}^{b_j{\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)}_j}}{e^{a_j{\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)}_j}+e^{b_j{\left({\mathit{A}}^t{\mathit{\lambda }}^t\right)}_j}}},j=1,\dots ,M.$$

(15)

Here, ${\mathit{\lambda }}^{*}\in {\mathbb{R}}^M$ is the point at which $\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})\equiv ⟨\mathit{\lambda },\mathit{y}⟩-M\left({\mathit{A}}^t\mathit{\lambda }\right)$ achieves its maximum value. Moreover,

$$\mathrm{\Psi }\left({\mathit{\xi }}^{*}\right)=\mathrm{\Sigma }({\mathit{\lambda }}^{*},\mathit{y}).$$

(16)

Proof. 

To obtain the solution (15), we use the standard Lagrange multipliers technique to minimize $\mathrm{\Psi }\left(\mathit{\xi }\right)$, where we first define the Lagrangian function

$$L(\mathit{\xi },\mathit{\lambda })=\mathrm{\Psi }\left(\mathit{\xi }\right)-⟨\mathit{\lambda },(\mathit{A}\mathit{\xi }-\mathit{y})⟩.$$

Then, equating ${\nabla }_{\mathit{\xi }}L=0$ and ${\nabla }_{\mathit{\lambda }}L=0$, one obtains the following system for ${\mathit{\xi }}^{*},{\mathit{\lambda }}^{*}$:

$$\begin{array}{c}{\nabla }_{\mathit{\xi }}\mathrm{\Psi }\left({\mathit{\xi }}^{*}\right)={\mathit{A}}^t{\mathit{\lambda }}^{*},\end{array}$$

(17)

$$\begin{array}{c}\mathit{A}{\mathit{\xi }}^{*}=y.\end{array}$$

(18)

We know that ${\nabla }_{\mathit{\xi }}\mathrm{\Psi }$ is invertible; therefore, from (17) and (14), we obtain

$${\mathit{\xi }}^{*}={\left({\nabla }_{\mathit{\xi }}\mathrm{\Psi }\right)}^{-1}\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)={\nabla }_{\mathit{\tau }}M\left(\mathit{\tau }\right),$$

with $\mathit{\tau }={\mathit{A}}^t{\mathit{\lambda }}^{*}$. The equality (16) follows from Corollary 3.3.11 in [14]. □

Keep in mind that the first K components of ${\mathit{\xi }}^{*}$ solve problem (3) and the last N are the estimated measurement error ${ϵ}^{*}$. Moreover, notice that, since $\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})=⟨\mathit{\lambda },\mathit{y}⟩-M\left({\mathit{A}}^t\mathit{\lambda }\right)$ is a strictly convex, infinitely differentiable function, its maximizer occurs at ${\mathit{\lambda }}^{*}$ satisfying

$${\nabla }_{\mathit{\lambda }}\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})=0\iff \mathit{A}{\mathit{\xi }}^{*}=y$$

with ${\mathit{\xi }}^{*}$ given by (15). In addition, since most numerical software packages are written to solve a minimization problem by default, instead of maximizing $\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})$, it is convenient to minimize $-\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})$. (see https://metrumresearchgroup.github.io/bbr/ (accessed on 23 January 2025)) for an example. This combines the usual gradient method with a step reduction procedure at each iteration. This is convenient because the objective function may be very flat near the minimum.)

The Reconstruction Error

When one solves problem (8) or its variational version (15) numerically, the solution ${\mathit{\xi }}^{*}$ need not satisfy $\mathit{A}{\mathit{\xi }}^{*}=\mathit{y}$ exactly. The reconstruction error simply measures how large the offset is with respect to the problem data. In methods that use ${\parallel \mathit{A}\mathit{\xi }-\mathit{y}\parallel }^2,$ the value of the objective function at the optimum, namely $\parallel \mathit{A}{\mathit{\xi }}^{*}{-\mathit{y}\parallel }^2,$ is, at the same time, a measure of the reconstruction error. In our approach, the minimum value $\mathrm{\Psi }\left({\mathit{\xi }}^{*}\right)$ does not measure the quality of the reconstruction error. Nevertheless, we know from Theorem 1 that

$$\parallel {\nabla }_{\mathit{\lambda }}\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})\parallel =\parallel \mathit{A}{\mathit{\xi }}^{*}-\mathit{y}\parallel .$$

(19)

So, at the end, the quantitative reconstruction is the same. However, as our method allows us to estimate the additive noise, which is given by the last N components of ${\mathit{\xi }}^{*},$ let us call this vector ${\mathit{ϵ}}^{*},$ and an estimate of the additive measurement error is given by $\parallel {\mathit{ϵ}}^{*}\parallel =\parallel {\mathit{A}}_0{\mathit{x}}^{*}-\mathit{y}\parallel .$

As we do not make any assumption about the statistical nature of the measurement noise, if it is supposed that the components of ${\mathit{ϵ}}^{*}$ are a sample of some random variable, and the number of data points is large, one could use the output of our method combined with the standard statistical methodology to determine the distribution of the noise affecting the measurement process.

3. Numerical Examples

Here, we consider several examples. The first set of examples is built from one taken from the textbook by Stone ([4], Ch. 8), which we examine from several points of view to exemplify the difference between our method and the least squares optimization procedure. We consider various cases depending on whether we assume that there is an experimental error or not and depending on whether the number of data points is larger or smaller than the number of unknown parameters. After these variations, we present a simulated example along the lines suggested in the Introduction for the R package v. 4.4.1 CVXR by [15].

All examples are programmed in R. We use CVXR to implement CVX with the ECOS_BB solver (https://github.com/embotech/ecos, accessed on 23 January 2025), which is a branch-and-bound procedure for the solution of mixed-integer convex second-order cone problems. For our optimization method, we use the spectral projected gradient method for large-scale optimization with simple constraints and Barzilai–Borwein step length strategies [16] (R package BB [17]).

3.1. A Textbook Example

To motivate the least squares method, Ref. [4] presents data from a chemical engineering process for polymerization. The data consist of 6 runs of coded temperatures, $h=(-5,-3,-1,1,3,5)$, and the corresponding values of the process, given in the following vector:

$${\mathit{y}}^t=(58.9,71.6,78.0,81.6,86.0,85.2)$$

(20)

The transpose of the matrix ${\mathit{A}}_0$ is

$${\mathit{A}}_0^t=\left(\begin{array}{cccccc}-5& -3& -1& 1& 3& 5\\ 25& 9& 1& 1& 9& 25\\ 1& 1& 1& 1& 1& 1\end{array}\right)$$

(21)

In fact, the original linear regression problem has a design matrix without the middle row. Since the solution obtained is poor, a non-linear regression of order two is suggested. So, suppose that the underlying model is $y=x_3+x_{1}h+x_2{h}^2.$ (We have changed the notations relative to [4] to follow ours.) Our first task is to estimate the ranges for

$$x_3=y\left(0\right),x_1={\displaystyle \frac{dy}{dh}}\left(0\right),x_2={\displaystyle \frac{1}{2}}{\displaystyle \frac{d^{2}y}{d{h}^2}}\left(0\right).$$

Here, we consider the following possibility: the mid-point for the ranges of $x_1$ and $x_2$ is taken to be the average of the numerical computation of these quantities. The consecutive incremental quotients of y are

$${\displaystyle \frac{\Delta y}{\Delta h}}=(6.35,3.2,1.8,2.2,-0.4)\Rightarrow {\displaystyle \frac{1}{5}}\sum {\displaystyle \frac{\Delta y}{\Delta h}}=2.63$$

Note that the quotients decrease, suggesting that the curve bends down (that it may be concave), except that there could be a measurement error at the fourth or fifth data point. The average of the incremental quotients of the incremental quotients yields a mid-point for the range of $x_2,$

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\Delta \left(\frac{\Delta y}{\Delta h}\right)}{\Delta h}}& =& (-1.57,-0.7,0.2,-1.3)\hfill \\ & \Rightarrow & {\displaystyle \frac{1}{2}}\left({\displaystyle \frac{1}{4}}\sum {\displaystyle \frac{\Delta \left(\frac{\Delta y}{\Delta h}\right)}{\Delta h}}\right)=-0.421.\hfill \end{array}$$

The mid-point for the range of the intercept is taken to be the average of $y\left(1\right)$ and $y(-1)$, which is $79.8.$ To be conservative, we allow for a $50\%$ miscalculation of the quantities above and obtain that the ranges for $x_1$ and $x_2$ are as in Table 1, whereas that for $x_3$ is obtained by adding (and subtracting) to the mid-point the length of one step with the estimated slope at $h=0.$

Table 1. Ranges for the unknowns.

${\mathit{x}}_1$	(1.32, 3.94)
$x_2$	(−0.63, −0.21)
$x_3$	(77.1, 82.5)

After this preamble, we are ready to consider several possible cases and comparisons.

3.1.1. Data Without Errors

We solve problem (3), $\mathit{y}={\mathit{A}}_0\mathit{x}$, with $\mathit{y}$ given as the vector (20), ${\mathit{A}}_0$ a $6\times 3$ matrix, whose transpose is shown in (21), and $\mathit{x}=(x_1,x_2,x_3)$ a solution constrained to the box $\mathcal{K}=[1.32,3.94]\times [-0.63,-0.21]\times [77.1,82.5]$. Our convex optimization method gives solution $\hat{\mathit{x}}=(2.547,-0.337,80.815)$, which, not surprisingly, is similar to the one obtained by the least squares method in [4] and also similar to solutions obtained by optimization methods such as CVXR [7,15], which are based on minimizing $\parallel {\mathit{A}}_0{\mathit{x}-\mathit{y}\parallel }^2$ (see our comments on consistency in Section 4 below). Substituting these estimated coefficients in the second-order polynomial model and evaluating for different h, we can compare our estimated values $\hat{y}$ to the given data values y in Table 2.

Table 2. Estimated values $\hat{y}$ vs. registered values y.

h	−5	−3	−1	1	3	5
y	58.9	71.6	78.0	81.6	86.0	85.2
$\widehat{y}$	59.65	70.14	77.93	83.02	85.42	85.12

3.1.2. Data with Errors

We assume now that the data have measurement errors. We take $d=20$ to set an ample box for the error (i.e., to be within $[-20,20]$). So, this time, we use our method to solve problem (7), $\mathit{y}={\mathit{A}}_e\mathit{x}$, with $\mathit{y}$ as before and the extended $6\times 9$ matrix ${\mathit{A}}_e=\left[{\mathit{A}}_0{\mathbb{I}}_6\right]$. The solution $\mathit{x}$ is constrained to the same box as in the previous section, and, after solving, we obtain the estimated values $\widehat{\mathit{x}}=(2.611,-0.403,79.976)$. Inserting these into the polynomial model, we obtain the estimated values $\widehat{y}$ for different h. Our method also gives an estimation of the error $\widehat{\mathit{ϵ}}$, and we show all of these quantities, together with the data values, in Table 3.

Table 3. Estimated values $\widehat{y}$ and errors vs. registered values y.

h	−5	−3	−1	1	3	5
y	58.9	71.6	78.0	81.6	86.0	85.2
$\widehat{y}$	56.84	68.52	76.96	82.18	84.18	82.95
$\widehat{\mathit{ϵ}}$	2.06	3.08	1.04	−0.58	1.82	2.23

One can see that the estimated $\widehat{y}$ plus the estimated error $\widehat{\mathit{ϵ}}$ sum up to the true data value.

Note that the extended problem $\mathit{y}={\mathit{A}}_e\mathit{\xi }$ is under-determined and always ill-posed since $N<K+N$ (in our example, $N=6$, $K=3$). Since ${\mathit{A}}_e=\left[{\mathit{A}}_0{\mathbb{I}}_N\right],$ then

$${\mathit{A}}_e^t{\mathit{A}}_e=\left(\begin{array}{cc}{\mathit{A}}_0^t{\mathit{A}}_0& {\mathit{A}}_0^t\\ {\mathit{A}}_0& {\mathbb{I}}_N.\end{array}\right)$$

The determinant of this block matrix vanishes since

$$\mathit{det}\left({\mathit{A}}_e^t{\mathit{A}}_e\right)=\det \left({\mathbb{I}}_N\right)\mathit{det}({\mathit{A}}_0^t{\mathit{A}}_0-{\mathit{A}}_0^t{\mathbb{I}}_N{\mathit{A}}_0)=0.$$

Therefore, as mentioned in Section 4, this enhances the convenience of our approach as it does not require inverting matrices or the quest for generalized inverses. We note that, in this case, a method like CVX also succeeds in finding the solution by applying an interior point method, which works well for small and medium-sized problems, but, for a larger problem, it switches to a first-order solver, which can be slow if the problem is not well conditioned [15]. In this particular example, CVXR yields the estimated coefficients ${\widehat{\mathit{x}}}_{\mathit{cvx}}=(2.364,-0.466,80.974)$; however, the mean square error of the estimated values of y with these coefficients, namely ${\widehat{y}}_{\mathit{cvx}}$, with respect to the observed values is $\mathit{mse}({\widehat{y}}_{\mathit{cvx}}-y)=4.688$, while, for the estimate obtained with our method, we have $\mathit{mse}(\widehat{y}-y)=3.921$. We shall give further proof of the superior precision of our method with respect to CVX in Section 3.2.

3.1.3. Case in Which the Data Are Scarce

We now assume that it is very difficult or costly to measure different values of $\mathit{y}$ and that we only have two values, ${\mathit{y}}^t=(81.6,85.2)$, corresponding to $h=1,5$, respectively. The transpose of the matrix ${\mathit{A}}_0$ is

$${\mathit{A}}_0^t=\left(\begin{array}{cc}1& 5\\ 1& 25\\ 1& 1\end{array}\right)$$

(22)

We keep the bounds for the solution the same as in the previous problems. Note that the problem $\mathit{y}={\mathit{A}}_0\mathit{x}$ is ill-posed since $N<K$ ($N=2$, $K=3$). Then, solving while disregarding errors with our method, we obtain $\widehat{\mathit{x}}=(2.967,-0.344,78.977)$. Solving the same problem with the CVXR method yields the solution ${\widehat{\mathit{x}}}_{\mathit{cvx}}=(3.735,$$-0.473,$$78.337)$. In Table 4, we present the original data values (for all h), the estimations $\widehat{y}$ given by the solution with our method, and the estimations ${\widehat{y}}_{\mathit{cvx}}$ obtained with the solution given by CVXR.

Table 4. Estimated values $\widehat{y}$ and ${\widehat{y}}_{\mathit{cvx}}$ and registered values y.

h	−5	−3	−1	1	3	5
y	58.9	71.6	78.0	81.6	86.0	85.2
$\widehat{y}$	55.53	66.97	75.66	81.6	84.77	85.2
${\widehat{y}}_{\mathit{cvx}}$	47.84	62.87	74.13	81.6	85.29	85.2

Here, our method performs better than CVX when the data are scarce. The values of $\widehat{\mathit{x}}$ yield a better estimate of the unobserved values of $\mathit{y}$. Comparing the mean square errors for both methods and the observed data, we have $\mathit{mse}(\widehat{y}-y)=178.39$ and $\mathit{mse}({\widehat{y}}_{\mathit{cvx}}-y)=286.65$.

3.2. A Simulated Example

For this set of examples, we generated Gaussian data with N observations and K predictors, for different values of N and K. Thus, our design matrix $\mathit{A}$ is of size $N\times K$, with entries obtained by generating $NK$ random numbers from a $\mathcal{N}(0,1)$ random variable. The outcome values are generated by $\mathit{y}=\mathit{A}\mathit{x}+\mathit{e},$ where $\mathit{e}\in {\mathbb{R}}^N$ is a sample from a $\mathcal{N}(0,0.05)$ random variable, and the regression coefficients form an arithmetic progression of difference 1 from $\lceil 1-K/2\rceil$ to $\lceil K/2\rceil$ (e.g., for $K=10$, $\mathit{x}=(-4,-3,-2,-1,0,1,2,3,4,5)$). The goal is to recover the values of $\mathit{x}$. We assume that the experimenter has certain knowledge of the range of values for the coefficients and the error, so she sets the constraint boxes as $[-\lceil K/2\rceil -1,\lceil K/2\rceil +1]$ for the coefficients and $[-1,1]$ (i.e., parameter $d=1$) for the error.

We tested this linear system for various values of N and K, considering $N<K$, $N=K$, or $N>K$. Note that, due to the random nature of $\mathit{A}$ and $\mathit{e}$, for each pair of values of $(N,K)$, we repeat the reconstruction experiment 100 times and report the average of the square of the norm of the respective errors, namely ${\parallel \mathit{e}\parallel }^2={\parallel \mathit{y}-\mathit{A}\mathit{x}\parallel }^2$ for our method and $\parallel {\mathit{e}}_{\mathit{cvx}}{\parallel }^2$ for the CVX reconstruction. Moreover, for large values of N and K, it is convenient to scale down the design matrix $\mathit{A}$ and the error $\mathit{e}$ (e.g., multiply by ${10}^{-2}$ for the cases of $N\ge 200$). This is to prevent the exploding of the exponentials in the objective function $\mathrm{\Sigma }(\mathit{\lambda },\mathit{y})$, which could disrupt the convergence of the underlying solver.

Table 5 shows the square of the norms of the reconstructions resulting from the application of our method and that of the CVX method, for various combinations of $(N,K)$. We see that, in all cases, our method excels CVX in accuracy, with around a 70–90% improvement in most cases.

Table 5. Comparison of the norms of the reconstruction errors for CVX, $\parallel {\mathit{e}}_{\mathit{cvx}}\parallel$, and our method, $\parallel \mathit{e}\parallel$, for different values of N and K. The results are averaged over 100 runs.

N	K	${\parallel \mathit{e}\parallel }^2$	$\parallel {\mathit{e}}_{\mathit{cvx}}{\parallel }^2$
20	10	$3.3\times {10}^{-2}$	$1.8\times {10}^{-1}$
20	20	$8.6\times {10}^{-2}$	$5.7\times {10}^{-1}$
20	40	$9.8\times {10}^{-4}$	$1.8\times {10}^{-1}$
100	100	0.06	1.15
200	100	0.32	1.19
200	300	$1.4\times {10}^{-3}$	$9\times {10}^{-1}$
300	200	1.17	10.47
300	300	2.63	11.32
300	500	$3.5\times {10}^{-3}$	$2.3$

4. A Consistency Issue

Here, we add a few consistency remarks. A natural question is the following: what does the method yield when $\mathit{A}$ is invertible? Let us apply the standard method of Lagrange multipliers to solve problem (8):

$$\mathrm{Find}{\mathit{\xi }}^{*}=\mathit{argmin}\{\mathrm{\Psi }\left(\mathit{\xi }\right):\mathit{\xi }\in \mathcal{K},\mathrm{and}\mathit{A}\mathit{\xi }=\mathit{y}\}.$$

Form the Lagrangian, $\mathrm{\Psi }\left(\mathit{\xi }\right)-⟨\mathit{\lambda },(\mathit{A}\mathit{\xi }-\mathit{y})⟩,$ and we equate its gradient with respect to $\mathit{\xi }$ to 0 to obtain

$${\nabla }_{\xi }\mathrm{\Psi }\left({\mathit{\xi }}^{*}\right)-{\mathit{A}}^t{\mathit{\lambda }}^{*}=0.$$

We know that ${\nabla }_{\mathit{\xi }}\mathrm{\Psi }$ is invertible; therefore,

$${\mathit{\xi }}^{*}={\left({\nabla }_{\xi }\mathrm{\Psi }\right)}^{-1}\left({\mathit{A}}^t\mathit{\lambda }\right).$$

Consider now the dual problem

$$\mathrm{Find}{\mathit{\lambda }}^{*}=\mathit{argmax}\{⟨\mathit{\lambda }\mathit{y}⟩-M\left({\mathit{A}}^t\mathit{\lambda }\right)\}.$$

The first-order condition for ${\mathit{\lambda }}^{*}$ to be the maximizer is that

$$\mathit{A}{\nabla }_{\tau }M\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)=\mathit{y}\iff {\nabla }_{\tau }M\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)={\mathit{A}}^{-1}\mathit{y}.$$

Now, invoking (14) and combining the two identities, we obtain that

$${\mathit{\xi }}^{*}={\left({\nabla }_{\mathit{\xi }}\mathrm{\Psi }\right)}^{-1}\left({\mathit{A}}^t{\mathit{\lambda }}^{*}\right)={\mathit{A}}^{-1}\mathit{y}.$$

Similarly, when $\mathit{A}$ is of full rank and ${\left({\mathit{A}}^t\mathit{A}\right)}^{-1}$ exists, a variation of the previous theme yields that ${\left({\mathit{A}}^t\mathit{A}\right)}^{-1}{\mathit{A}}^t\mathit{y}$ is a solution to ${\mathit{A}}^t\mathit{A}\mathit{\xi }={\mathit{A}}^t\mathit{y}$. The matrix ${\left({\mathit{A}}^t\mathit{A}\right)}^{-1}{\mathit{A}}^t$ is a generalized inverse of $\mathit{A}.$

Clearly, this is not surprising, but it is useful to know that the consistency is maintained in these particular cases. Once more, the strong features of the duality approach, even when $\mathit{A}$ is invertible but ill-conditioned, is very important, as there is no need to invert matrices to numerically find the maximizer of the dual problem.

5. Final Remarks

To summarize, the main features of our approach are as follows: we define a strictly convex function whose domain is the constraint set and whose minimization provides an explicit representation of the solution to problems (3) or its extended version (7). The extended version can be viewed as a regularization of the original problem, designed to address cases where the data do not lie in the range of the operator. Additionally, the approach ensures the tractability of the dual problem for the Lagrange multipliers, which is particularly advantageous from a numerical perspective, as highlighted in the final paragraph of the previous section.

Pending issues to address are as follows. Our approach does not impose any specific assumptions about the measurement errors, besides the fact that they lie within a bounded interval. It would be interesting to explore scenarios where a large number of measurements are feasible, using the output to determine the statistical properties of the measurement error. As demonstrated in Section 3.1.3, the method performs effectively even when data are scarce, but it would be worthwhile to investigate the asymptotic properties of the estimators.

Finally, by employing appropriate vectorization, we can derive a stylized version of the problem addressed in [18] using the method of maximum entropy in the mean. The problem consists of finding an $M\times K$ matrix $\mathit{X}$ and an $N\times K$ matrix $\mathit{E}$ that satisfy

$$\mathit{Y}={\mathit{A}}_0\mathit{X}+\mathit{E}.$$

(23)

Here, ${\mathit{A}}_0$ is a given $N\times M$ matrix, and $\mathit{Y}$ is a given $N\times K$ matrix, and it is also required that the components of $\mathit{X}$ and $\mathit{E}$ satisfy some box constraint. It can be proven that, after vectorization, this problem reduces to a problem equal to (7).