Selected Applications of the Theory of Connections: A Technique for Analytical Constraint Embedding

Abstract

In this paper, we consider several new applications of the recently introduced mathematical framework of the Theory of Connections (ToC). This framework transforms constrained problems into unconstrained problems by introducing constraint-free variables. Using this transformation, various ordinary differential equations (ODEs), partial differential equations (PDEs) and variational problems can be formulated where the constraints are always satisfied. The resulting equations can then be easily solved by introducing a global basis function set (e.g., Chebyshev, Legendre, etc.) and minimizing a residual at pre-defined collocation points. In this paper, we highlight the utility of ToC by introducing various problems that can be solved using this framework including: (1) analytical linear constraint optimization; (2) the brachistochrone problem; (3) over-constrained differential equations; (4) inequality constraints; and (5) triangular domains.

1. Introduction

The Theory of Connections (ToC), introduced by D. Mortari [1] (see also [2,3,4]), is a framework that transforms constrained problems into unconstrained problems by deriving “constrained expressions”. Using these expressions, the governing physical models are modified and solved in the space without constraints. Below, we present a brief overview. For univariate functions, Ref. [1] introduced the constrained expression using the form,

$${\displaystyle y(x)=g(x)+\sum _{k=1}^n{\eta }_k{h}_k(x),}$$

(1)

where the $h_k(x)$ are n assigned linearly independent functions, $g(x)$ is a free function, and the coefficients ${\eta }_k$ are derived by imposing the n constraints. The resulting constrained expression always satisfies the constraints, as long as $g(x)$ is defined, where the constraints are defined.

As an example, we consider a function $y(x)$ subject to the constraints $y(x_1)=y_1$ and $y^{\prime }(x_2)=y_2^{\prime }$. Following Equation (1), and selecting $h_1(x):=1$ and $h_2(x):=x$, the following two expressions are obtained [3,4].

$$\begin{array}{cc}\hfill y_1& =g_1+{\eta }_1+{\eta }_2{x}_1\hfill \\ \hfill y_2^{\prime }& =g_2^{\prime }+{\eta }_2\hfill \end{array}$$

where $g_1$ and $g_2^{\prime }$ are values of the free function and its derivatives evaluated at the two constraint points $x_1$ and $x_2$. The unknowns ${\eta }_1$ and ${\eta }_2$ can be solved leading to,

$$\begin{array}{cc}\hfill {\eta }_1& =y_1-g_1-(y_2^{\prime }-g_2^{\prime })x_1\hfill \\ \hfill {\eta }_2& =y_2^{\prime }-g_2^{\prime }.\hfill \end{array}$$

Plugging the $\eta$ coefficients back into Equation (1) and simplifying, the final constrained expression is produced,

$$y(x)=g(x)+(y_1-g_1)+(y_2^{\prime }-g_2^{\prime })(x-x_1).$$

It can be seen that the constraints are always satisfied ($y(x_1)=y_1$ and $y^{\prime }(x_2)=y_2^{\prime }$), as long as $g(x_1)$ and $g^{\prime }(x_2)$ are defined. This is an example that demonstrates Equation (1). The theory is general and can be used for many other constraints. In the paper, we will present other examples (see also [1,3] for more examples).

In general, the ${\eta }_k$ coefficients are expressed in terms of the independent variables and in terms of $g(x)$ evaluated where the constraints are defined. The Equation (1) is called constrained expression because it represents all possible functions satisfying the n constraints, regardless of the function $g(x)$. Please note that $g(x)$ can be discontinuous, partially defined, and even the Dirac delta function, as long as there are no constraints specified where the delta function is infinite.

The function $y(x)$ in Equation (1) typically is applied to ODEs or some variational problems. In general, in the example of ODEs, we can consider a problem for $y=y(x)$,

$$F(x,y,y^{\prime },\cdots ,y^{(n)})=0,$$

subject to the constraints, which can include various types of constraints, (e.g., constraints on the function or derivatives, integral constraints, or so on). Here, F is some equation of $y(x)$ and its derivatives. Using a ToC expression (Equation (1)), this equation is transformed to an unconstrained problem where the unknown parameter is defined by a new function $g=g(x)$. This transformation can be written as,

$$\tilde{F}(x,g,g^{\prime },\cdots ,g^{(n)})=0,$$

(2)

where $\tilde{F}$ is equations completely independent of the problem constraints. Moreover, $\tilde{F}$ differs from F and represents modified constitutive relations. The major advantage of Equation (2) is that it can be solved as an unconstrained optimization problem. Past work has focused on expanding the function of $g(x)$ by a linear basis [5,6,7] such that

$$g(x)={\mathit{\xi }}^{\mathrm{T}}\mathit{h}(x),$$

where $\mathit{h}(x)$ is a vector of global functions and $\mathit{\xi }$ is vector of unknown coefficients used in the optimization process. These coefficients are determined by minimizing the residual of Equation (2) at some collocation points [8,9]. In general, one can use locally supported $\mathit{h}(x)$ functions. One can also avoid a linear functional representation of $g(x)$ and use a non-linear representation, for example, neural networks and support vector machines [10].

The Multivariate Theory of Connections [2] extends the original univariate theory [1] to n-dimensions and to any-degree boundary constraints. This extension can be summarized by the expression,

$$y(\mathit{x})=A(c(\mathit{x}))+g(\mathit{x})-A(g(\mathit{x})),$$

where $\mathit{x}=\{x_1,x_1,\cdots ,x_n\}$ is the vector of n orthogonal coordinates, $c(\mathit{x})$ is a function specifying the boundary constraints, $A(c(\mathit{x}))$ is any interpolating function satisfying the boundary constraints, and $g(\mathit{x})$ is the free function. Several examples of constrained expressions can be found in Refs. [1,2].

The major research effort of Theory of Connections has been applied to solve linear [4] and non-linear [3] ODEs. This has been done by expanding the free function, $g(x)$, in terms of a set of basis functions (e.g., orthogonal polynomials, Fourier, etc.). Linear or iterative non-linear least-squares is then used to solve for the coefficients of this expansion. This approach to solve ODEs/PDEs has many advantages over traditional methods: (1) it consists of a unified framework to solve IVP, BVP, or multi-values problems, (2) it provides an approximated solution expressed via analytical functions that can be used for subsequent algebraic manipulation, (3) the solution accuracy is usually obtained fast for many application problems, (4) the procedure can be numerically robust (very low condition number), and (5) it can solve ODE/PDE subject to a variety of constraint types: absolute, relative, linear, non-linear, and integral. Additionally, this technique has recently been extended to solve 2-dimensional PDEs [10].

The purpose of this paper is to introduce a set of five new applications of Theory of Connections; applications that are not covered in the previous references and where ToC is found effective. The main motivation and a short rational description of the new applications considered in this paper are:

• Analytic linear constraint optimization. In this application, the analytical problem of finding the extreme of a quadratic vectorial form subject to linear constraint is obtained using ToC instead of using the classical Lagrange multipliers technique.
• Brachistochrone problem. This application is dedicated to one of the most famous problem in calculus of variations: the brachistochrone problem [11]. This problem is solved numerically and at machine error accuracy via the two-point boundary-value problem of the Euler-Lagrange equation associated with the brachistochrone functional integral definition. In particular, the solution is obtained by deriving the differential equation in polar coordinates.
• Over-constrained differential equations. This application takes into consideration the problem of solving ODEs subject to more constraints than the degree of the ODE which arise in many areas of science and engineering. For example, in the problem of orbit determination of a satellite the number of measurements exceeds the order of the governing differential equation [12]. Furthermore, multi-purpose optimization [13,14,15] deals specifically with optimizing across multiple objective functions simultaneous where trade-offs (weighting) between two conflicting objectives must be taken into account. In this section, a weight least-squares solution is provided for over-constrained differential equations by assigning relative weights to the constraints and then solving the weighted constrained ODE by ToC. Additionally, another example showing the sequence of continuous solutions of an IVP morphing into an BVP is provided.
• Inequality constraints. This application extends the ToC framework to include inequality constraints [16]. This is obtained using a combination of sigmoid functions to keep the constrained expression within the inequality constraints. This allows for the derivation of functions constrained by user-defined bounds than can be asymmetric, continuous, or symmetric discontinuous functions. The main motivation for this constraint type comes from optimal control problems (bounded control inputs).
• Triangular domains. Validated by mathematical proof, Ref. [2] has extended the original univariate theory [1] to the multivariate theory [2]. This extension represents the multivariate formulation of ToC subject to arbitrary-order derivative constraints in rectangular domains. This provides an analytical procedure to obtain constrained expressions in any orthogonal/rectangular space that can be used to transform constrained problems into unconstrained problems. In Ref. [2] particular emphasis and details are given to the 2-dimensional case, because of the most important applications to surfaces (PDEs, Topography, Visualization, etc.). This application begins the extension of ToC to triangular domains by deriving the surfaces satisfying boundary constraints.

The complete solution derivations of each one of these new applications, validated by numerical examples, are detailed in the subsequent sections.

2. Analytic Linear Constraints Optimization

This section shows an analytical application of the Theory of Connections. The problem, known as quadratic programming (QP) subject to equality constraints, consists of deriving a closed-form solution of the problem to find the min/max of the quadratic function, $F(\mathit{x})$, subject to $m<n$ linear constraints,

$$F(\mathit{x})={\displaystyle \frac{1}{2}}{\mathit{x}}^{\mathrm{T}}Q\mathit{x}+{\mathit{c}}^{\mathrm{T}}\mathit{x}\mathrm{subject}\mathrm{to}:A\mathit{x}=\mathit{b},$$

(3)

where $Q\in {\mathbb{R}}^{n\times n}$ and $Q=Q^{\mathrm{T}}>0$, $\mathit{c}\in {\mathbb{R}}^n$, $A\in {\mathbb{R}}^{m\times n}$, and $\mathit{b}\in {\mathbb{R}}^m$, are all assigned, and D has rank m (all rows are independent). Let us search a solution in the form,

$$\mathit{x}=\mathit{g}+A^{\mathrm{T}}\mathit{\eta },$$

The constraint implies,

$$A(\mathit{g}+A^{\mathrm{T}}\mathit{\eta })=\mathit{b}\to A{A}^{\mathrm{T}}\mathit{\eta }=\mathit{b}-A\mathit{g},$$

and

$$\mathit{\eta }={(A{A}^{\mathrm{T}})}^{-1}\mathit{b}-{(A{A}^{\mathrm{T}})}^{-1}A\mathit{g}.$$

Therefore,

$$\mathit{x}=\mathit{g}+A^{\mathrm{T}}{(A{A}^{\mathrm{T}})}^{-1}\mathit{b}-A^{\mathrm{T}}{(A{A}^{\mathrm{T}})}^{-1}A\mathit{g}={\mathit{x}}_0+\left[I_{n\times n}-A^{\mathrm{T}}{(A{A}^{\mathrm{T}})}^{-1}A\right]\mathit{g}={\mathit{x}}_0+D\mathit{g}.$$

Substituting this expression of $\mathit{x}$ in Equation (3) we obtain,

$$F(\mathit{g})={\displaystyle \frac{1}{2}}{({\mathit{x}}_0+D\mathit{g})}^{\mathrm{T}}Q({\mathit{x}}_0+D\mathit{g})+{\mathit{c}}^{\mathrm{T}}({\mathit{x}}_0+D\mathit{g}).$$

The problem of finding the extreme of Equation (3) is now an unconstrained optimization problem. Stationary condition implies

$${\displaystyle \frac{\mathrm{d}f(\mathit{g})}{\mathrm{d}\mathit{g}}}=0\to \begin{array}{|c|}\hline \mathcal{A}\mathit{g}+\mathit{d}=0\\ \hline\end{array}$$

(4)

where

$$\mathcal{A}={\displaystyle \frac{1}{2}}{D}^{\mathrm{T}}(Q+Q^{\mathrm{T}})D\mathrm{and}\mathit{d}={\displaystyle \frac{1}{2}}{D}^{\mathrm{T}}(Q+Q^{\mathrm{T}}){\mathit{x}}_0+D^{\mathrm{T}}\mathit{c}.$$

In this example, we show how to transform a constrained optimization problem to unconstrained optimization problem and find the unconstrained $g(x)$, which is the solution of Equation (4). This shows a simple application of Theory of Connections. Unfortunately, the matrix $\mathcal{A}$ is singular. However, Equation (4) is a consistent linear system. Therefore, the solution can be found in the null space of $\mathcal{A}$. This is computed by the Moore-Penrose inverse matrix (pseudo-inverse). Let $\mathcal{A}=C\mathsf{\Lambda }{C}^{\mathrm{T}}$ be the spectral decomposition of $\mathcal{A}$, where the eigenvector matrix, C, is an orthogonal matrix because matrix $\mathcal{A}$ is symmetric. Therefore, the solution of the problem given in Equation (3) is,

$$\begin{array}{|c|}\hline \mathit{x}={\mathit{x}}_0-DC{\mathsf{\Lambda }}^{\ast }{C}^{\mathrm{T}}\mathit{d},\\ \hline\end{array}$$

where the diagonal matrix ${\mathsf{\Lambda }}^{\ast }$ contains the inverse of the nonzero eigenvalues and zero for zero eigenvalues.

3. Brachistochrone Problem

The brachistochrone problem is one of many problems arising from the calculus of variations. This body of work focuses on finding the extrema of functionals (mappings from functions to real numbers). Associated functions that maximize or minimize the functionals can be found using the Euler-Lagrange equation. In general, this equation produces an ODE which when solved produces the extremal function. The main application of ToC to the calculus of variation is solving the specific ODEs which can be highly complex.

Consider finding the function, $f_s(x)$, minimizing the integral,

$${\displaystyle I(f_s)=min{\int }_a^{b}F(x,f(x),f^{\prime }(x))\mathrm{d}x}$$

(5)

where $x\in [a,b]$, $f(x)$ is an unknown function satisfying $f(a)=f_a$ and $f(b)=f_b$, and $F(x,f(x),f^{\prime }(x))$ is the functional. An extremal, $f_0(x)$, of the integral given in Equation (5) is the solution of the Euler-Lagrange equation,

$${\displaystyle \frac{\partial F}{\partial f}}\left(x,f_0(x),f_0^{\prime }(x)\right)={\displaystyle \frac{\mathrm{d}}{\mathrm{d}x}}\left[{\displaystyle \frac{\partial F}{\partial f^{\prime }}}\left(x,f_0(x),f_0^{\prime }(x)\right)\right].$$

(6)

The solution of this equation, $f_0(x)$, is an extremal solution (where the derivative is zero, which is a necessary condition). This solution, however, is not sufficient as $f_0(x)$ may be associated with some local minima, maxima, or saddle points. In other word, after solving Equation (6), we can consider $f_s(x)=f_0(x)$, if the condition,

$${\displaystyle {\int }_a^{b}F(x,f_0(x),f_0^{\prime }(x))\mathrm{d}x\le min{\int }_a^{b}F(x,f(x),f^{\prime }(x))\mathrm{d}x,\forall f(x)}$$

is verified. This can be done, for example, by evaluating the second derivative (Hessian) of $F(x,f(x),f^{\prime }(x))$. Figure 1 shows three different coordinate systems that may be used to solve the brachistochrone problem.

Typically, the brachistochrone problem is written using coordinate system (a) of Figure 1. In this coordinate system, the total travel time of the sliding bead is given by the functional [11],

$$F(x,y,y^{\prime })=\sqrt{{\displaystyle \frac{1+y^{\prime 2}}{2gy}}},y=y(x)$$

(7)

subject to,

$$y(0)=0\mathrm{and}y(x_1)=y_1.$$

Using the steps of the calculus of variations outlined in Equation (6) on the function in Equation (7) yields the differential equation for the function $y=y(x)$,

$$2y{y}^{″}+y^{\prime 2}+1=0.$$

(8)

However, this coordinate system produces the case that $y^{\prime }(0)=\infty$ at the initial point and cannot be solved using ToC. If the axes are oriented as shown in coordinate system (b) of Figure 1, the functional used in the calculus of variations becomes,

$$F(x,y,y^{\prime })=\sqrt{\frac{1+y^{\prime 2}}{2gx}},y=y(x)$$

which after performing the steps of the calculus of variations leads to the differential equation for the function $y=y(x)$,

$$2x{y}^{″}+y^{\prime 3}+y^{\prime }=0.$$

(9)

The problem with the differential equation given in Equation (9) is that the solution may take on two different values of $y(x)$ for a given value of x. Analytically, the issues with the DEs given in Equations (8) and (9) are circumvented by solving the problems parametrically. Unfortunately, the DE for the brachistochrone written in the parametric form either contains not-a-number or infinite values when implemented numerically, or admits a trivial solution. For example, let $x(p)$ and $y(p)$ each be functions of the parameter p and their derivatives with respect to p be given by the notation: $\mathrm{d}x/\mathrm{d}p=x_p$ and ${\mathrm{d}}^{2}x/\mathrm{d}{p}^2=x_{pp}$. Then, Equation (9) can be rewritten using the chain rule as

$$2x\frac{y_{pp}}{x_{pp}}+{\left(\frac{y_p}{x_p}\right)}^3+\frac{y_p}{x_p}=0,x=x(p),y=y(p).$$

(10)

Unfortunately, Equation (10) will have infinite values anytime $x_p=0$ or $x_{pp}=0$. Therefore, it cannot be implemented numerically. Alternatively, one could write the DE as

$$2x{y}_{pp}{x}_p^3+x_{pp}(y_p{x}_p^2+y_p^3)=0,x=x(p),y=y(p).$$

(11)

However, Equation (11) admits the trivial solution $x=0$. Attempts have been made to avoid the trivial solution by giving a starting value for the non-linear least-squares that is far from the solution $x=0$, but none of these attempts were successful.

Another option is to express the brachistochrone problem in polar coordinates (coordinate system (c) in Figure 1). If the coordinates are chosen correctly, then the problems of the Cartesian coordinate systems can be avoided, and the problem does not need to be parameterized. A downside of this approach is a significantly more complicated DE; however, by using the ToC method, solving this DE is very easy. Using this coordinate system, the energy of an object sliding without friction is given by,

$${\displaystyle \frac{1}{2}}m{v}^2=mgrsin\vartheta ,r=r(\vartheta ).$$

(12)

In addition, the differential path length in polar coordinates is given by,

$$\mathrm{d}s=\sqrt{r^2+r^{\prime 2}}\mathrm{d}\vartheta ,$$

(13)

where $r^{\prime }=\mathrm{d}r/\mathrm{d}\vartheta$. Furthermore, using Equation (12), the velocity, v, can be written as,

$$v=\frac{\mathrm{d}s}{\mathrm{d}t}=\sqrt{2grcos\vartheta }.$$

(14)

Using Equations (13) and (14) one can express the total time to get from the start point to the end point as

$$T={\int }_0^t\mathrm{d}t={\int }_{{\vartheta }_0}^{{\vartheta }_f}\frac{\sqrt{r^2+r^{\prime 2}}}{\sqrt{2grcos\vartheta }}\mathrm{d}\vartheta .$$

(15)

Let the integrand of Equation (15) be defined as the functional $F(\vartheta ,r(\vartheta ),r^{\prime }(\vartheta ))$. Then, the calculus of variations can be used to minimize the travel time. This leads to the DE, after some simplification, given in Equation (16).

$$-r^2(\dot{r}tan\vartheta +2\ddot{r})-tan\vartheta {\dot{r}}^3+3\dot{r}r+r^3=0,r=r(\vartheta ).$$

(16)

Using the differential equation given by Equation (16), the ToC method was used to solve the particular case with initial condition $(x_0,y_0)=(0,0)$ and final condition $(x_f,y_f)=(1,5)$. Figure 2 shows information associated with solving the brachistochrone problem and highlights the error between the ToC solution and the real solution. Additionally, the residuals of the DE are between the order of ${10}^{-12}$ and ${10}^{-16}$ when the ToC formulation is used and $g(x)$ is expressed by 60 Chebyshev polynomials. It can be seen that this method can estimate the solution of the brachistochrone problem with high accuracy.

4. Over-Constrained Differential Equations

The seminal paper on ToC [1] presented Equation (1) as a way to incorporate n constraints to form a constrained expression, this equation is repeated below:

$${\displaystyle y(x)=g(x)+\sum _{k=1}^n{\eta }_k{h}_k(x).}$$

This process leads to a ($n\times n$) matrix populated by the values of $h_k(x)$ evaluated at the constraint conditions. The values of ${\eta }_k$ can then be solve by simply inverting this matrix. Yet, through this process, it is not required that number of $\eta$ coefficients equals the number of applied constraints, and thus constraint matrix can also be written for n values of $\eta$ and m constraints. Which leads to,

$$M\mathit{\eta }=\left[\begin{array}{cccc}{h}_1^{(d_1)}(x_1)& h_2^{(d_1)}(x_1)& \cdots & h_n^{(d_1)}(x_1)\\ h_1^{(d_2)}(x_2)& h_2^{(d_2)}(x_2)& \cdots & h_n^{(d_2)}(x_2)\\ ⋮& ⋮& \ddots & ⋮\\ h_1^{(d_m)}(x_m)& h_2^{(d_m)}(x_m)& \cdots & h_n^{(d_m)}(x_m)\end{array}\right]\left\{\begin{array}{c}{\eta }_1\\ {\eta }_2\\ ⋮\\ {\eta }_n\end{array}\right\}=\left\{\begin{array}{c}{y}_1^{(d_1)}-g_1^{(d_1)}\\ y_2^{(d_2)}-g_2^{(d_2)}\\ ⋮\\ y_m^{(d_m)}-g_m^{(d_m)}\end{array}\right\},$$

where $d_i$ denotes the order of derivative of the constraint at point $x_i$ where $i\in [1,m]$. This becomes an over-determined system with the constraint matrix M being $(m\times n)$. The $\eta$ values can still be solved by incorporating a weighted least-squares technique, where we define W as a diagonal weighting matrix,

$$W:=\left[\begin{array}{cccc}{w}_1& 0& \cdots & 0\\ 0& w_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & w_m\end{array}\right],$$

leading to the solution,

$$\mathit{\eta }={(M^{\mathrm{T}}WM)}^{-1}{M}^{\mathrm{T}}W\left\{\begin{array}{c}{y}_1^{(d_1)}-g_1^{(d_1)}\\ y_2^{(d_2)}-g_2^{(d_2)}\\ ⋮\\ y_m^{(d_m)}-g_m^{(d_m)}\end{array}\right\}.$$

Unlike the traditional ToC approach where the $\eta$ values produced an equation always satisfying the given constraints, the weight least-squares approach produces a weighted constrained expression which satisfies the constraints in a weighted (relative) sense. To exactly solve a Differential Equation, the number of constraints MUST be equal to the order of the Differential Equation. Yet, through the ToC method, the constrained expression is formed and then the differential equation is solved. This separation allows for the incorporation of more constraints than the order of the Differential Equation. The concept of this is highlighted in Figure 3 which provides an outline which distinguishes the ToC approach from classical methods in interpolation and least-squares. Part (c) in Figure 3 shows the original formulation of ToC where the number of the $\eta$ coefficients was equal to the number of constraints incorporated. Part (d) highlights the work of this section which combines this general interpolation method with a weighted least-squares technique for the constraints.

4.1. Second Order Differential Equation with Three Point Constraints. A Numerical Example

As a numerical test, let us consider solving the Differential Equation,

$$y^{″}+2y^{\prime }+y=0,y=y(x).$$

(17)

For this example, let us assume that a dynamical system described by Equation (17) is “observed” at three points $x=[-1,-0.5,+1]$ and these measurements are subject to normally distributed noise such that

$$y(-1)=\mathcal{N}(y_1^{true},{\sigma }_1^2),y(-0.5)=\mathcal{N}(y_2^{true},{\sigma }_2^2),\mathrm{and}\mathcal{N}(y_3^{true},{\sigma }_3^2),$$

where for this problem $y_1^{true}=5$, $y_2^{true}=4.515$, and $y_3^{true}=2$. For this, we define the weight matrix based on the variances,

$$W:=\left[\begin{array}{ccc}{w}_1& 0& 0\\ 0& w_2& 0\\ 0& 0& w_3\end{array}\right]=\left[\begin{array}{ccc}{\sigma }_1^{-2}& 0& 0\\ 0& {\sigma }_2^{-2}& 0\\ 0& 0& {\sigma }_3^{-2}\end{array}\right].$$

Using the development in the prior section, this Differential Equation can incorporate information from all three observations even though the Differential Equation is only second-order. Since the Differential Equation is second-order, the solution must be searched by the following constrained expression,

$$y(x)=g(x)+{\eta }_1{h}_1(x)+{\eta }_2{h}_2(x).$$

Then the values of ${\eta }_1$ and ${\eta }_2$ are computed by weighted least-squares where we define $h_1(x):=1$ and $h_2(x):=x$,

$$\left\{\begin{array}{c}{\eta }_1\\ {\eta }_2\end{array}\right\}=\underset{V}{\underbrace{{\left(M^{\mathrm{T}}WM\right)}^{-1}{M}^{\mathrm{T}}W}}\left\{\begin{array}{c}{y}_1-g_1\\ y_2-g_2\\ y_3-g_3\end{array}\right\},M:=\left[\begin{array}{cc}\hfill 1& x_1\\ \hfill 1& x_2\\ \hfill 1& x_3\end{array}\right]$$

where the M matrix is defined by imposing the constraints and the W matrix is populated by the scaled variances above. The solution leads to a constrained expression of the form,

$$y(x)=g(x)+{\beta }_1(x)(y_1-g_1)+{\beta }_2(x)(y_2-g_2)+{\beta }_3(x)(y_3-g_3),$$

(18)

such that

$$\begin{array}{ccc}\hfill {\beta }_1(x)& =& w_1\left[w_2(x_2-x){\Delta }_{21}+w_3(x_3-x){\Delta }_{31}\right]/({\Delta }_{21}^2{w}_1{w}_2+{\Delta }_{31}^2{w}_1{w}_3+{\Delta }_{32}^2{w}_2{w}_3)\hfill \\ \hfill {\beta }_2(x)& =& w_2\left[w_1(x-x_1){\Delta }_{21}+w_3(x_3-x){\Delta }_{32}\right]/({\Delta }_{21}^2{w}_1{w}_2+{\Delta }_{31}^2{w}_1{w}_3+{\Delta }_{32}^2{w}_2{w}_3)\hfill \\ \hfill {\beta }_3(x)& =& w_3\left[w_1(x-x_1){\Delta }_{31}+w_2(x-x_2){\Delta }_{32}\right]/({\Delta }_{21}^2{w}_1{w}_2+{\Delta }_{31}^2{w}_1{w}_3+{\Delta }_{32}^2{w}_2{w}_3).\hfill \end{array}$$

such that ${\Delta }_{ij}:=x_i-x_j$. Equation (18) represents all functions satisfying constraints relative to the given variances ${\sigma }_1^2$, ${\sigma }_2^2$, and ${\sigma }_3^2$. With the constraints completely captured in Equation (18), the $g(x)$ function represents the solution space that satisfies the three constraints by least-squares. Now let us move toward solving Equation (17). First, let the free function $g(x)$, be expressed as a linear combination of a given basis set where $g(x)={\mathit{\xi }}^{\mathrm{T}}\mathit{h}(x)$. Substituting this into Equation (18) leads to,

$$y(x)={\mathit{\xi }}^{\mathrm{T}}\mathit{h}(x)+{\beta }_1(x)(y_1-{\mathit{\xi }}^{\mathrm{T}}{\mathit{h}}_1)+{\beta }_2(x)(y_2-{\mathit{\xi }}^{\mathrm{T}}{\mathit{h}}_2)+{\beta }_3(x)(y_3-{\mathit{\xi }}^{\mathrm{T}}{\mathit{h}}_3),$$

where the constrained expression can be reduced to the form,

$$y(x)={\mathit{\xi }}^{\mathrm{T}}\mathit{a}(x)+b(x),$$

This form now has transformed the constrained optimization problem into an unconstrained optimization problem on $\mathit{\xi }$, and can be solved using the techniques developed in Ref. [3,4]. In this specific problem, a Monte Carlo simulation of 10,000 trials was conducted to determine space that the function $y(x)$ could occupy given the “observation” uncertainty and subject to dynamics governed by the Differential Equation. Figure 4 highlights this solution. In this figure, plot (a) shows the Differential Equation solution space given the observation uncertainty and plot (b) highlights the residuals of the Differential Equation over the entire simulation. It can be seen that the residuals of all solutions are between ${10}^{-13}$ and ${10}^{-14}$. Plots (c), (d), and (e) display the distribution of the constraint points around the true value. Note: these values are sampled from the solutions of the Differential Equation and not the constraints specified in the constrained expression.

Through this test, a probability bound for the Differential Equation can be produced, along with an estimated mean. For all solutions, the residual of the differential remained less than ${10}^{-13}$ verifying the accuracy of the method. Additionally, an interesting result of this test is in the final estimated solutions. Most evident in the constraints at $x=-0.5$ and $x=+1$, the $3\sigma$ of the Differential Equation is less than that of the observation $3\sigma$. This arises because the loss function in the ToC method minimizes the residuals of the Differential Equation, and now in the over-constrained ToC method the residuals are minimized simultaneously with the weighted least-squares of the observations (or constraints).

4.2. Initial to Boundary-Value Problem Transformation. A Numerical Example

Another application made possible by this extension is a continuous transformation of an initial value problem to a boundary-value problem. The following numerically validates this approach. Consider solving,

$$y^{″}+\left[cos(3x^2)-3x+1\right]y^{\prime }+\left[6sin(4x^2)-e^{cos(3x)}\right]y=2{\displaystyle \frac{[1-sin(3x)](3x-\pi )}{4-x}},y=y(x),$$

subject to the three constraints

$$y_0=y(-1)=-2,y_0^{\prime }=y^{\prime }(-1)=-2,\mathrm{and}{y}_f=y(+1)=2.$$

Since the Differential Equation is second-order, the constrained expression takes the form,

$$y(x)=g(x)+{\eta }_1{h}_1(x)+{\eta }_2{h}_2(x).$$

Since the highest order constraint is the first derivative, we can safely define $h_1(x):=1$ and $h_2(x):=x$. Then the values of ${\eta }_1$ and ${\eta }_2$ are computed again by weighted least-squares,

$$\left\{\begin{array}{c}{\eta }_1\\ {\eta }_2\end{array}\right\}=\underset{V}{\underbrace{{\left(M^{\mathrm{T}}WM\right)}^{-1}{M}^{\mathrm{T}}W}}\left\{\begin{array}{c}{y}_0-g_0\\ y_0^{\prime }-g_0^{\prime }\\ y_f-g_f\end{array}\right\},$$

where the M matrix is defined by imposing the constraints and the W matrix is a diagonal matrix of weights, i.e.,

$$M:=\left[\begin{array}{cc}1& -1\\ 0& 1\\ 1& 1\end{array}\right]\mathrm{and}W:=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1-\alpha & 0\\ 0& 0& \alpha \end{array}\right],$$

and $\alpha$ is a weight parameter such that $\alpha \in [0,1]$, which is synonymous with the transformation from the IVP to the BVP. From this derivation, V becomes

$$V={\displaystyle \frac{1}{1+4\alpha -{\alpha }^2}}\left[\begin{array}{ccc}\alpha +1& {(1-\alpha )}^2& 1+2\alpha -{\alpha }^2\\ -2\alpha & 1-{\alpha }^2& 2\alpha \end{array}\right].$$

and solving for $\mathit{\eta }$ leads again Equation (18) of the form,

$$y(x)=g(x)+{\beta }_1(x)(y_1-g_1)+{\beta }_2(x)(y_2-g_2)+{\beta }_3(x)(y_3-g_3),$$

where the $\beta$ terms are defined by,

$$\begin{array}{ccc}\hfill {\beta }_1(x)& =& (1+\alpha [1-2x]/(1+4\alpha -{\alpha }^2)\hfill \\ \hfill {\beta }_2(x)& =& ({[1-\alpha ]}^2+[1-{\alpha }^2]x)/(1+4\alpha -{\alpha }^2)\hfill \\ \hfill {\beta }_3(x)& =& (1+2\alpha [1+x]-{\alpha }^2)/(1+4\alpha -{\alpha }^2).\hfill \end{array}$$

Again, $g(x)$ can be defined as $g={\mathit{\xi }}^{\mathrm{T}}\mathit{h}(x)$ and the unknown coefficient vector $\mathit{\xi }$ can be solved with least-squares. Figure 5 shows this transformation “surface” along with the residuals of the Differential Equation for validation of the method. It can be seen that the mean residual overall $\alpha$ values are on the order of ${10}^{-14}$ with a standard deviation on the same order. This plot shows the solution of the Differential Equation, $y(x)$, continuously morphing from initial point and derivative constraints to initial and final point constraints.

5. Inequality Constraints

In some cases, it may be desired to find all possible trajectories between two functions, $f_u(x)$ and $f_d(x)$, that represent upper and lower continuous bounds, respectively. This means that $f_u(x)>f_d(x)$, $\forall x\in [x_{min},x_{max}]$. All trajectories between these two functions can be approximated by the following expression,

$$y(x)=g(x)+[f_u(x)-g(x)]\sigma (k,g(x)-f_u(x))+[f_d(x)-g(x)]\sigma (k,f_d(x)-g(x))$$

(19)

where $g(x)$ represents the free function. In the cases that there are also equality constraints to be applied, the function $g(x)$ can be replaced by Equation (1). Additionally,

$$\sigma (k,z)={\displaystyle \frac{1}{1+e^{-kz}}}\mathrm{and}{\displaystyle \frac{\mathrm{d}\sigma (k,z)}{\mathrm{d}z}}=k\sigma (k,z)\left(1-\sigma (k,z)\right),$$

is a sigmoid function with smoothing parameter k. The sigmoid function acts as a “switching function” that subtracts off the portion of $g(x)$ that exceeds the bounds. In this equation, k is not infinite, and thus $y(x)$ will experience some overshooting/undershooting when $g(x)>f_u(x)$ or $g(x)<f_d(x)$ that must be quantified. To analyze this, let us consider a simple case where the function is only constrained from above with $f_u(x)$ and therefore only overshooting exists.

$$y(x)=g(x)+[f_u(x)-g(x)]\sigma (k,g(x)-f_u(x)).$$

(20)

For this case, the overshooting phenomenon is shown in Figure 6.

Since the only free part of the equation is $g(x)$, the maximum value of the entire function, $y(x)$, will be when the derivative ($\mathrm{d}y(x)/\mathrm{d}g(x)=0)$. Performing this derivative, Equation (20) becomes,

$$\frac{\mathrm{d}y}{\mathrm{d}g}=0=1-\frac{1}{1+e^{-k(g(x)-f_u(x))}}-\frac{e^{-k(g(x)-f_u(x))}k(f_u(x)-g(x))}{{(1+e^{-k(g(x)-f_u(x))})}^2},$$

which can be solved for $g(x)$. The resulting value, $g{(x)}_{\max }$, is the value of $g(x)$ that will cause the maximum overshoot in $y(x)$. This process yields,

$$g{(x)}_{max}=\frac{k{f}_u(x)+\mathrm{productlog}(e^{-1})+1}{k}$$

where productlog is the product logarithm function (also-called the omega function or the Lambert W function). Substituting this result back into Equation (19), we can obtain the maximum value that $y(x)$ can take.

$$y{(x)}_{max}=f_u(x)+\frac{\mathrm{productlog}(e^{-1})}{k}.$$

(21)

Now, we use Equation (21) to solve for a modified upper bound, ${\widehat{f}}_u(x)$, such that the maximum value of $y(x)$ is $f_u(x)$: ${\widehat{f}}_u(x)$ is the adjusted boundary shown in Figure 6. This yields Equation (22).

$${\widehat{f}}_u(x)=f_u(x)-\frac{\mathrm{productlog}(e^{-1})}{k}.$$

(22)

A similar derivation can be made for the lower bound ${\widehat{f}}_d(x)$; the solution is shown in Equation (23).

$${\widehat{f}}_d(x)=f_d(x)+\frac{\mathrm{productlog}(e^{-1})}{k}.$$

(23)

The functions ${\widehat{f}}_u(x)$ and ${\widehat{f}}_d(x)$ replace $f_u(x)$ and $f_d(x)$ in Equation (19), resulting in,

$$y(x)=g(x)+[{\widehat{f}}_u(x)-g(x)]\sigma (k,g(x)-{\widehat{f}}_u(x))+[{\widehat{f}}_d(x)-g(x)]\sigma (k,{\widehat{f}}_d(x)-g(x)).$$

(24)

Equation (21) shows that the value of k is the only user-selected parameter that effects the difference between $y{(x)}_{max}$ and $f_u(x)$, because $\mathrm{productlog}(e^{-1})$ is a constant. Increasing the value of k will decrease the difference between $y{(x)}_{max}$ and $f_u(x)$, and also the difference between $y{(x)}_{min}$ and $f_d(x)$. Decreasing the value of k will have the opposite effect. Equation (24) was used to generate Figure 7, which contains randomly generated inequality boundaries and functions. This figure shows that all the random functions stay within the randomly generated inequality boundaries, and thus demonstrates how this technique can be used to enforce inequality boundary constraints.

6. Triangular Domains

This section demonstrates how to construct a constrained expression with a free function, $g(x,y)$, that will always meet the Dirichlet boundary conditions on a triangular domain. This section is broken down into two subsections. The first proves that an affine transformation exists that can transform the unit triangle into any desired triangle: shown in Figure 8. The second subsection shows how to build the constrained expression for the unit triangle, and how to extend it, via the affine transformation given in the first subsection, to any triangular domain.

6.1. Affine Transformation from the Unit Triangle to the Generic Triangle

Consider a unit triangle, which is shown on the left plot of Figure 8. This unit triangle has the vertices defined by the vectors, ${\mathit{P}}_1:={\{0,0\}}^{\mathrm{T}}$, ${\mathit{P}}_2:={\{0,1\}}^{\mathrm{T}}$, and ${\mathit{P}}_3:={\{1,0\}}^{\mathrm{T}}$.

This subsection presents an affine transformation, denoted by $\mathcal{A}$, that transforms the coordinates of a unit triangle defined in the $[X,Y]$ (capital frame) into the coordinates $[x,y,z]$ of the triangle in the desired coordinate frame.

Let the three vectors ${\mathit{P}}_i$, where $i\in \{1,2,3\}$, be the position vectors of vertices of the unit triangle in the starred frame. Similarly, ${\mathit{p}}_i$, where $i\in \{1,2,3\}$, be the position vectors of the vertices of the desired triangle in the regular frame. Then, the affine coordinate transformation given in Equation (25) transforms the unit triangle in the capital frame into the desired triangle in the desired frame.

$$\left\{\begin{array}{c}x\\ y\\ z\end{array}\right\}=\left[\begin{array}{ccc}{\mathit{p}}_3-{\mathit{p}}_1,& {\mathit{p}}_2-{\mathit{p}}_1,& ({\mathit{p}}_3-{\mathit{p}}_1)\times ({\mathit{p}}_2-{\mathit{p}}_1)\end{array}\right]\left\{\begin{array}{c}X\\ Y\\ 0\end{array}\right\}+{\mathit{p}}_1=A\left\{\begin{array}{c}X\\ Y\\ 0\end{array}\right\}+{\mathit{p}}_1.$$

(25)

The following identities prove the mapping provided by Equation (25),

$${\mathit{p}}_1=A\left\{\begin{array}{c}0\\ 0\\ 0\end{array}\right\}+{\mathit{p}}_1,{\mathit{p}}_2=A\left\{\begin{array}{c}0\\ 1\\ 0\end{array}\right\}+{\mathit{p}}_1,\mathrm{and}{\mathit{p}}_3=A\left\{\begin{array}{c}1\\ 0\\ 0\end{array}\right\}+{\mathit{p}}_1.$$

The inverse transformation is also of interest, and can be written as,

$$\left\{\begin{array}{c}X\\ Y\end{array}\right\}={\left[\begin{array}{ccc}{\mathit{p}}_3-{\mathit{p}}_1,& {\mathit{p}}_2-{\mathit{p}}_1,& ({\mathit{p}}_3-{\mathit{p}}_1)\times ({\mathit{p}}_2-{\mathit{p}}_1)\end{array}\right]}^{-1}\left(\left\{\begin{array}{c}x\\ y\\ z\end{array}\right\}-{\mathit{p}}_1\right).$$

(26)

The inverse transformation will always exist if the triangle is non-degenerate. The first column of A is the vector that points from ${\mathit{p}}_3$ to ${\mathit{p}}_1$, and the second column is the vector from that points from ${\mathit{p}}_2$ to ${\mathit{p}}_1$. For a triangle, these two vectors will never be parallel. The third column of A is defined as the cross-product of the first two columns, and will therefore be orthogonal to the vectors that make up the first two columns. Therefore, the columns of A are linearly independent, and A is invertible. Therefore, the inverse transform always exists.

6.2. Coons-Type Surface on the Unit Triangle

Consider unit triangle with boundary constraints provided by the boundary functions, $F_1(0,Y)$, $F_2(X,1-X)=F_2(1-Y,Y)$ when $Y+X=1$, and $F_3(X,0)$. For this unit triangle, the following Coons-type surface can be constructed,

$$\begin{array}{cc}\hfill F(X,Y)=& (1-X-Y)\left[F_3(X,0)+F_1(0,Y)-F_1(0,0)\right]+\hfill \\ & +X\left[F_3(X,0)-F_3(1-Y,0)+F_2(1-Y,Y)\right]+\hfill \\ & +Y\left[F_1(0,Y)-F_1(0,1-X)+F_2(X,1-X)\right].\hfill \end{array}$$

(27)

From Equation (27) one can show that

$$F(X,0)=F_3(X,0),F(0,Y)=F_1(0,Y),\mathrm{and}\left\{\begin{array}{cc}F(X,1-X)=\hfill & F_2(X,1-X)\hfill \\ F(1-Y,Y)=\hfill & F_2(1-Y,Y).\hfill \end{array}\right.$$

Let’s write Equation (27) in the following compact form,

$$F(X,Y)=C(F_1,F_2,F_3)$$

(28)

where $C(F_1,F_2,F_3)$ is a function of the boundary constraints, $F_1(0,Y)$, $F_2(X,1-X)=F_2(1-Y,Y)$, and $F_3(X,0)$.

6.3. ToC Surfaces on the Unit Triangle

Equation (28) can be specified for any surface, $Z=G(X,Y)$. This means that we can write the identity,

$$G(X,Y)=C(G_1,G_2,G_3)$$

where $G_1=G(0,Y)$, $G_2=G(X,1-X)$ or $G_2=G(1-Y,Y)$, and $G_3=G(X,0)$. Therefore, ToC allows us to write the constrained expression representing all surfaces satisfying the boundary conditions on the unit triangle as,

$$F(X,Y)=C(F_1,F_2,F_3)+G(X,Y)-C(G_1,G_2,G_3)$$

(29)

where $G(X,Y)$ is a free function. The constrained expression given in Equation (29) can be written explicitly as,

$$\begin{array}{cc}\hfill F(X,Y)=& G(X,Y)+(1-X-Y)\left[F_3(X,0)-G(X,0)+F_1(0,Y)-G(0,Y)-F_1(0,0)+G(0,0)\right]+\hfill \\ & X\left[F_3(X,0)-G(X,0)-F_3(1-Y,0)+G(1-Y,0)+F_2(1-Y,Y)-G(1-Y,Y)\right]+\hfill \\ & Y\left[F_1(0,Y)-G(0,Y)-F_1(0,1-X)+G(0,1-X)+F_2(X,1-X)-G(X,1-X)\right].\hfill \end{array}$$

6.4. ToC Surfaces on the Generic Triangle

The ToC surfaces on the generic triangle can be derived using Equation (29) and the affine mapping given in Equation (26). Setting, $\mathit{X}=(X,Y)$, Equation (29) can be rewritten as

$$F(\mathit{X})=C[F_1(\mathit{X}),F_2(\mathit{X}),F_3(\mathit{X})]+G(\mathit{X})-C[G_1(\mathit{X}),G_2(\mathit{X}),G_3(\mathit{X})].$$

Setting, $\mathit{x}=(x,y,z)$, the affine transformation can be written as

$$\mathit{X}=A^{-1}\left(\mathit{x}-{\mathit{p}}_1\right),$$

and the ToC Surfaces on the Generic Triangle can be written as,

$$F(\mathit{x})=C[F_1(\mathit{X}(\mathit{x})),F_2((\mathit{X}(\mathit{x}))),F_3((\mathit{X}(\mathit{x}))]+G((\mathit{X}(\mathit{x})))-C[G_1((\mathit{X}(\mathit{x}))),G_2((\mathit{X}(\mathit{x}))),G_3((\mathit{X}(\mathit{x})))].$$

Figure 9 and Figure 10 show two ToC surfaces for a generic triangle with the same boundary conditions but different free functions, $g(x)$. The boundary conditions used to create these figures were,

$$\begin{array}{cc}\hfill F_1& =(6-y)(y-1.5)cos(3x)+4,\hfill \\ \hfill F_2& =(2-x)(x-3)exp(x)cos(2y)+4,\hfill \\ \hfill F_3& =(y-1.5)(x-3)\sqrt{xy}+4,\hfill \end{array}$$

and the vertices of the triangle were $P_1=(1,1.5)$, $P_2=(2,6)$, and $P_3=(3,1)$.

Figure 9 and Figure 10 show that the ToC surfaces meet the triangular boundary constraints (shown in black) regardless of the free function, $g(x,y)$, chosen.

7. Conclusions

This work demonstrated some of the applications of ToC as a unified framework. The first application showed how to solve QP problems subject to equality constraints without the use of Lagrange multipliers. Using ToC, the equality constraints are embedded into a constrained expression that transforms the problem into an unconstrained optimization problem. The second application showed how to solve one of the most famous calculus of variations problems, the brachistochrone problem, with ToC. The calculus of variations technique was used to derive a highly non-linear differential equation in polar coordinates, to avoid singularities, which was solved via ToC. The third application explored solving over-constrained differential equations, where there are more constraints than the order of the differential equation. Using ToC through weighted least-squares, we transform the problem into unconstrained problem, which can easily be solved. The fourth application showed how to extend ToC to incorporate inequality constraints by using sigmoid functions. In addition, a discussion on avoiding undershoot/overshoot was included. The fifth and final application showed how to write constrained expressions for triangular domains subject to Dirichlet boundary conditions.

Future work will investigate how some of these applications can be extended to solve optimal control problems. The QP problem subject to linear equality constraints and solving the brachistochrone problem represent the first research thrust in that direction, as these are some of the classic problems in the field of optimal control. Moreover, future work will investigate how these applications can be combined with the application on inequality constraints to solve optimal control problems subject to inequality constraints.

Furthermore, future work will investigate applications of over-constrained differential equations. One example is applications in orbit determination where measurements exceed the order of the differential equation governing the dynamics of the orbiting body. By using the approach developed for weighted least-squares solutions of differential equations, the dynamics of the body and variance of observational measurements can be combined into a single optimization problem. Another area of future work includes multi-objective optimization problems.

Additionally, future research will look to extend constrained expressions on triangular domains to include constraints on the derivatives normal to the sides of the triangle. The obvious application of this research extension will be in solving differential equations on triangular domains; however, this research extension also has applications in meshing, domain discretization, and visualization.