# The Multivariate Theory of Functional Connections: Theory, Proofs, and Application in Partial Differential Equations

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Univariate TFC

#### 2.1. Univariate Example # 1: Constraints at a Point

**Property**

**1.**

- Choose k support functions, ${s}_{k}(x)$.
- Write each switching function as a linear combination of the support functions with unknown coefficients.
- Based on the switching function definition, write a system of equations to solve for the unknown coefficients.

#### 2.2. Univariate Example # 2: Linear Constraints

**Property**

**2.**

## 3. General Formulation of the Univariate TFC

**Definition**

**1.**

**Definition**

**2.**

**Definition**

**3.**

**Definition**

**4.**

**Definition**

**5.**

**Definition**

**6.**

^{i}, is a linear operator that when operating on a function returns the function evaluated at the i-th specified constraint.

**Property**

**3.**

**Property**

**4.**

**Definition**

**7.**

**Theorem**

**1.**

**Proof.**

**Theorem**

**2.**

**Proof.**

**Theorem**

**3.**

**Proof.**

^{i}is a linear operator,

## 4. Multivariate TFC

#### 4.1. Recursive Application of Univariate TFC

^{1}f is then used as the free function in a constrained expression that includes all the constraints on ${x}_{2}$ to produce the expression

^{2}f. This method carries on until the final independent variable, ${x}_{n}$, is reached and the expression

^{n}f = f and is the multivariate constrained expression.

#### 4.1.1. Multivariate Example # 1: Value and Derivative Constraints

#### 4.1.2. Multivariate Example # 2: Linear Constraints

#### 4.1.3. Multivariate Constrained Expression Theorems

**Theorem**

**4.**

**Proof.**

**Theorem**

**5.**

**Proof.**

**Theorem**

**6.**

**Proof.**

#### 4.2. Tensor Form

- The elements of the first order sub-tensors of $\mathcal{M}$ acquired by setting all but one index equal to one are a zero followed by the projection functionals for the dimension associated with that index. Mathematically,$${\mathcal{M}}_{1\cdots {i}_{k}\cdots 1}=\left\{\begin{array}{cccc}0,& {}^{k}\rho {{}_{}^{}}_{1},& \cdots ,& {}^{k}\rho {{}_{}^{}}_{{\ell}_{k}}\end{array}\right\},$$
- The remaining elements of the $\mathcal{M}$ tensor, those that have more than one index not equal to one, are the geometric intersection of the associated projection functionals multiplied by a sign (− or +). Mathematically, this can be written as,$${\mathcal{M}}_{{i}_{1}{i}_{2}\cdots {i}_{n}}={}^{j}\u212d{}_{}^{{i}_{j}-1}\left(\right)open="["\; close="]">{}^{j}\u212d{}_{}^{{i}_{k}-1}\left(\right)open="["\; close="]">\cdots \left(\right)open="["\; close="]">{}^{h}\rho {{}_{}^{}}_{{i}_{h}-1}$$$${\mathcal{M}}_{{i}_{1}{i}_{2}\cdots {i}_{n}}={}^{h}\u212d{}_{}^{{i}_{h}-1}\left(\right)open="["\; close="]">{}^{j}\u212d{}_{}^{{i}_{j}-1}\left(\right)open="["\; close="]">\cdots \left(\right)open="["\; close="]">{}^{k}\rho {{}_{}^{}}_{{i}_{k}-1}$$

#### 4.2.1. Multivariate Example # 1: Value and Derivative Constraints Using the Tensor Form

#### 4.2.2. Multivariate Example # 2: Linear Constraints Using the Tensor Form

## 5. Applications to PDEs

#### 5.1. Defining the Free Function $g(\mathit{x})$

#### 5.2. Derivatives of the Free Function

#### 5.3. Discretization

#### 5.4. Summary of the Major Steps to Solving PDEs

## 6. Results

^{®}Core™ i7 and with 16 GB of RAM. All run times were calculated using the default_timer function in the Python

`timeit`package. In all cases, matrix inversion was handled with NumPy’s

`pinv`function.

- Training
- (a)
- Estimate the solution of the PDE (i.e., determine the coefficients $\mathbf{\xi}$) using the TFC method with n points per independent variable and basis functions up to the to the m-th degree.
- (b)
- Maximum training error: Using the training set discretization, determine the absolute error of the estimated solution compared with the true solution and record the maximum value.

- Test
- (a)
- Using converged $\mathbf{\xi}$ parameters from the training phase, refine the discretization of domain with $n=100$ equally spaced points per dimension and evaluate estimated solution at these points.
- (b)
- Maximum test error: Using the test set discretization, determine the absolute error of the estimated solution compared with the true solution and record the maximum value.

#### 6.1. Problem # 1

#### 6.2. Problem #2

## 7. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## Abbreviations

BVP | Boundary Value Problem |

FEM | Finite Element Method |

ODE | Ordinary Differential Equation |

PDE | Partial Differential Equation |

TFC | Theory of Functional Connections |

X-TFC | Extreme Theory of Functional Connections |

## Appendix A. Orthogonal Polynomials

#### Appendix A.1. Chebyshev Orthogonal Polynomials

#### Appendix A.2. Legendre Orthogonal Polynomials

## Appendix B. Solution Times

#### Appendix B.1. Problem #1 Solution Times

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 0.46 | - | - | - | - | |

10 | 0.49 | 1.45 | - | - | - | |

15 | 0.61 | 1.88 | 4.90 | - | - | |

20 | 0.68 | 2.42 | 6.50 | 13.83 | - | |

25 | 0.92 | 3.93 | 8.23 | 18.07 | 35.63 | |

30 | 0.97 | 4.09 | 13.23 | 22.09 | 50.88 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 0.053 | - | - | - | - | |

10 | 0.119 | 0.128 | - | - | - | |

15 | 0.210 | 0.288 | 0.430 | - | - | |

20 | 0.399 | 0.584 | 0.824 | 1.114 | - | |

25 | 0.697 | 1.011 | 1.582 | 2.344 | 3.609 | |

30 | 1.045 | 1.871 | 2.909 | 4.683 | 8.164 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 0.77 | - | - | - | - | |

10 | 0.58 | 1.41 | - | - | - | |

15 | 0.56 | 1.82 | 4.70 | - | - | |

20 | 0.65 | 2.56 | 6.62 | 13.74 | - | |

25 | 0.86 | 3.15 | 7.93 | 17.71 | 35.61 | |

30 | 1.32 | 3.53 | 9.47 | 22.06 | 46.26 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 0.061 | - | - | - | - | |

10 | 0.111 | 0.164 | - | - | - | |

15 | 0.214 | 0.298 | 0.368 | - | - | |

20 | 0.394 | 0.613 | 0.863 | 1.108 | - | |

25 | 0.692 | 1.095 | 1.606 | 2.389 | 3.505 | |

30 | 1.031 | 1.689 | 2.765 | 4.696 | 7.063 |

#### Appendix B.2. Problem #2 Solution Times

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 1.76 | - | - | - | - | |

10 | 2.14 | 5.56 | - | - | - | |

15 | 2.38 | 7.79 | 27.30 | - | - | |

20 | 2.99 | 10.17 | 36.38 | 71.75 | - | |

25 | 3.90 | 13.00 | 42.40 | 93.11 | 144.5 | |

30 | 4.62 | 18.20 | 48.68 | 105.7 | 185.8 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 0.547 | - | - | - | - | |

10 | 1.905 | 1.384 | - | - | - | |

15 | 4.045 | 3.337 | 4.460 | - | - | |

20 | 7.460 | 6.753 | 9.771 | 10.97 | - | |

25 | 11.93 | 12.95 | 17.43 | 21.15 | 22.88 | |

30 | 18.94 | 21.98 | 28.90 | 38.92 | 45.82 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 1.64 | - | - | - | - | |

10 | 1.90 | 5.46 | - | - | - | |

15 | 2.24 | 10.63 | 21.52 | - | - | |

20 | 3.27 | 11.74 | 29.56 | 69.51 | - | |

25 | 3.79 | 13.82 | 36.32 | 91.89 | 145.1 | |

30 | 4.17 | 13.73 | 51.42 | 112.4 | 181.4 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 0.508 | - | - | - | - | |

10 | 1.684 | 1.330 | - | - | - | |

15 | 3.875 | 4.489 | 3.849 | - | - | |

20 | 7.284 | 7.882 | 8.078 | 10.56 | - | |

25 | 11.63 | 12.86 | 15.50 | 21.35 | 23.15 | |

30 | 17.57 | 18.44 | 28.46 | 40.53 | 45.60 |

## References

- Mortari, D. The Theory of Connections: Connecting Points. Mathematics
**2017**, 5, 57. [Google Scholar] [CrossRef] [Green Version] - Mortari, D. Least-Squares Solution of Linear Differential Equations. Mathematics
**2017**, 5, 48. [Google Scholar] [CrossRef] - Mortari, D.; Johnston, H.; Smith, L. High Accuracy Least-squares Solutions of Nonlinear Differential Equations. J. Comput. Appl. Math.
**2019**, 352, 293–307. [Google Scholar] [CrossRef] [PubMed] - Johnston, H.; Mortari, D. Least-squares Solutions of Boundary-value Problems in Hybrid Systems. arXiv
**2019**, arXiv:math.OC/1911.04390. [Google Scholar] - Furfaro, R.; Mortari, D. Least-squares Solution of a Class of Optimal Space Guidance Problems via Theory of Connections. ACTA Astronaut.
**2019**. [Google Scholar] [CrossRef] - Johnston, H.; Schiassi, E.; Furfaro, R.; Mortari, D. Fuel-Efficient Powered Descent Guidance on Large Planetary Bodies via Theory of Functional Connections. arXiv
**2020**, arXiv:math.OC/2001.03572. [Google Scholar] - Mai, T.; Mortari, D. Theory of Functional Connections Applied to Nonlinear Programming under Equality Constraints. In Proceeding of the 2019 AAS/AIAA Astrodynamics Specialist Conference, Portland, ME, USA, 11–15 August 2019. [Google Scholar]
- Johnston, H.; Leake, C.; Efendiev, Y.; Mortari, D. Selected Applications of the Theory of Connections: A Technique for Analytical Constraint Embedding. Mathematics
**2019**, 7, 537. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Mortari, D.; Leake, C. The Multivariate Theory of Connections. Mathematics
**2019**, 7, 296. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Leake, C.; Johnston, H.; Smith, L.; Mortari, D. Analytically Embedding Differential Equation Constraints into Least Squares Support Vector Machines Using the Theory of Functional Connections. Mach. Learn. Knowl. Extr.
**2019**, 1, 60. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Leake, C.; Mortari, D. Deep Theory of Functional Connections: A New Method for Estimating the Solutions of Partial Differential Equations. Mach. Learn. Knowl. Extr.
**2020**, 2, 4. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Schiassi, E.; Leake, C.; Florio, M.D.; Johnston, H.; Furfaro, R.; Mortari, D. Extreme Theory of Functional Connections: A Physics-Informed Neural Network Method for Solving Parametric Differential Equations. arXiv
**2020**, arXiv:cs.LG/2005.10632. [Google Scholar] - Leake, C.; Mortari, D. An Explanation and Implementation of Multivariate Theory of Functional Connections via Examples. In Proceeding of the AIAA/AAS Astrodynamics Specialist Conference, Portland, ME, USA, 11–15 August 2019. [Google Scholar]
- Ye, J.; Gao, Z.; Wang, S.; Cheng, J.; Wang, W.; Sun, W. Comparative Assessment of Orthogonal Polynomials for Wavefront Reconstruction over the Square Aperture. J. Opt. Soc. Am. A
**2014**, 31, 2304–2311. [Google Scholar] [CrossRef] [PubMed] - Dunkl, C.F.; Xu, Y. Orthogonal Polynomials of Several Variables, 2nd ed.; Encyclopedia of Mathematics and Its Applications; Cambridge University Press: Cambridge, UK, 2014. [Google Scholar] [CrossRef] [Green Version]
- Xu, Y. Multivariate Orthogonal Polynomials and Operator Theory. Trans. Am. Math. Soc.
**1994**, 343, 193–202. [Google Scholar] [CrossRef] - Langtangen, H.P. Computational Partial Differential Equations: Numerical Methods and Diffpack Programming; OCLC: 851766084; Springer: Berlin/Heidelberg, Germany, 2003. [Google Scholar]
- Lanczos, C. Applied Analysis. In Progress in Industrial Mathematics at ECMI 2008; Dover Publications, Inc.: New York, NY, USA, 1957; p. 504. [Google Scholar]
- Wright, K. Chebyshev Collocation Methods for Ordinary Differential Equations. Comput. J.
**1964**, 6, 358–365. [Google Scholar] [CrossRef] [Green Version] - Maclaurin, D.; Duvenaud, D.; Johnson, M.; Townsend, J. Autograd. 2013. Available online: https://github.com/HIPS/autograd (accessed on 1 July 2020).
- Baydin, A.G.; Pearlmutter, B.A.; Radul, A.A.; Siskind, J.M. Automatic Differentiation in Machine Learning: A Survey. J. Mach. Learn. Res.
**2018**, 18, 1–43. [Google Scholar] - Lagaris, I.E.; Likas, A.; Fotiadis, D.I. Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Netw.
**1998**, 9, 987–1000. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Mall, S.; Chakraverty, S. Single Layer Chebyshev Neural Network Model for Solving Elliptic Partial Differential Equations. Neural Process. Lett.
**2017**, 45, 825–840. [Google Scholar] [CrossRef] - Sun, H.; Hou, M.; Yang, Y.; Zhang, T.; Weng, F.; Han, F. Solving Partial Differential Equation Based on Bernstein Neural Network and Extreme Learning Machine Algorithm. Neural Process. Lett.
**2019**, 50, 1153–1172. [Google Scholar] [CrossRef] - Huang, G.B.; Zhu, Q.Y.; Siew, C.K. Extreme learning machine: Theory and applications. Neurocomputing
**2006**, 70, 489–501. [Google Scholar] [CrossRef]

**Figure 3.**Multivariate example # 2 constrained expression evaluated using $g(x,y)={x}^{2}cosy+sin(2x)$. The blue line signifies the constraint on $u(0,y)$, the black lines signify the derivative constraint on ${u}_{y}(x,0)$ and the magenta lines signify the relative constraint $u(x,0)=u(x,1)$. The linear constraint $u(1,y)+u(2,y)=ysin(\pi y)$ is not easily visualized, but is nonetheless satisfied by the constrained expression.

**Table 1.**Equivalence of number of basis function compared to degree of basis expansion for both Problems #1 and #2.

m | Number of Functions |
---|---|

5 | 17 |

10 | 62 |

15 | 132 |

20 | 227 |

25 | 347 |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 6.26 × ${10}^{-4}$ | - | - | - | - | |

10 | 5.53 × ${10}^{-4}$ | 1.20 × ${10}^{-10}$ | - | - | - | |

15 | 5.30 × ${10}^{-4}$ | 1.17 × ${10}^{-10}$ | 4.44 × ${10}^{-16}$ | - | - | |

20 | 5.20 × ${10}^{-4}$ | 1.16 × ${10}^{-10}$ | 5.00 × ${10}^{-16}$ | 4.44 × ${10}^{-16}$ | - | |

25 | 5.13 × ${10}^{-4}$ | 1.15 × ${10}^{-10}$ | 7.22 × ${10}^{-16}$ | 2.61 × ${10}^{-15}$ | 5.55 × ${10}^{-16}$ | |

30 | 5.09 × ${10}^{-4}$ | 1.14 × ${10}^{-10}$ | 6.66 × ${10}^{-16}$ | 8.88 × ${10}^{-16}$ | 3.22 × ${10}^{-15}$ |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 6.26 × ${10}^{-4}$ | - | - | - | - | |

10 | 5.53 × ${10}^{-4}$ | 1.20 × ${10}^{-10}$ | - | - | - | |

15 | 5.30 × ${10}^{-4}$ | 1.17 × ${10}^{-10}$ | 4.44 × ${10}^{-16}$ | - | - | |

20 | 5.20 × ${10}^{-4}$ | 1.16 × ${10}^{-10}$ | 5.55 × ${10}^{-16}$ | 4.44 × ${10}^{-16}$ | - | |

25 | 5.13 × ${10}^{-4}$ | 1.15 × ${10}^{-10}$ | 4.44 × ${10}^{-16}$ | 4.44 × ${10}^{-16}$ | 5.55 × ${10}^{-16}$ | |

30 | 5.09 × ${10}^{-4}$ | 1.14 × ${10}^{-10}$ | 4.44 × ${10}^{-16}$ | 4.44 × ${10}^{-16}$ | 5.55 × ${10}^{-16}$ |

**Table 4.**Comparison of maximum training and test error of TFC with current state-of-the-art techniques for Problem # 1.

Method | Training Set Maximum Error | Test Set Maximum Error |
---|---|---|

TFC | $2.22\times {10}^{-16}$ | $4.44\times {10}^{-16}$ |

X-TFC [12] | $3.8\times {10}^{-13}$ | $5.1\times {10}^{-13}$ |

FEM | $2\times {10}^{-8}$ | $1.5\times {10}^{-5}$ |

Reference [22] | $5\times {10}^{-7}$ | $5\times {10}^{-7}$ |

Reference [23] | - | $3.2\times {10}^{-2}$ |

Reference [24] | - | $2.4\times {10}^{-4}$ |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 1.03 × 10${}^{-1}$ | - | - | - | - | |

10 | 9.20 × 10${}^{-2}$ | 2.49 × 10${}^{-5}$ | - | - | - | |

15 | 9.03 × 10${}^{-2}$ | 1.54 × 10${}^{-5}$ | 4.34 × 10${}^{-9}$ | - | - | |

20 | 8.94 × 10${}^{-2}$ | 1.52 × 10${}^{-5}$ | 4.56 × 10${}^{-9}$ | 5.33 × 10${}^{-15}$ | - | |

25 | 8.88 × 10${}^{-2}$ | 1.50 × 10${}^{-5}$ | 4.53 × 10${}^{-9}$ | 2.72 × 10${}^{-15}$ | 4.44 × 10${}^{-16}$ | |

30 | 8.85 × 10${}^{-2}$ | 1.49 × 10${}^{-5}$ | 4.51 × 10${}^{-9}$ | 2.72 × 10${}^{-15}$ | 3.33 × 10${}^{-16}$ |

m | 5 | 10 | 15 | 20 | 25 | |
---|---|---|---|---|---|---|

n | ||||||

5 | 1.03 × 10${}^{-1}$ | - | - | - | - | |

10 | 9.20 × 10${}^{-2}$ | 2.49 × 10${}^{-5}$ | - | - | - | |

15 | 9.03 × 10${}^{-2}$ | 1.54 × 10${}^{-5}$ | 4.34 × 10${}^{-9}$ | - | - | |

20 | 8.94 × 10${}^{-2}$ | 1.52 × 10${}^{-5}$ | 4.56 × 10${}^{-9}$ | 5.33 × 10${}^{-15}$ | - | |

25 | 8.88 × 10${}^{-2}$ | 1.50 × 10${}^{-5}$ | 4.53 × 10${}^{-9}$ | 2.78 × 10${}^{-15}$ | 5.55 × 10${}^{-16}$ | |

30 | 8.85 × 10${}^{-2}$ | 1.49 × 10${}^{-5}$ | 4.51 × 10${}^{-9}$ | 2.73 × 10${}^{-15}$ | 5.55 × 10${}^{-16}$ |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Leake, C.; Johnston, H.; Mortari, D.
The Multivariate Theory of Functional Connections: Theory, Proofs, and Application in Partial Differential Equations. *Mathematics* **2020**, *8*, 1303.
https://doi.org/10.3390/math8081303

**AMA Style**

Leake C, Johnston H, Mortari D.
The Multivariate Theory of Functional Connections: Theory, Proofs, and Application in Partial Differential Equations. *Mathematics*. 2020; 8(8):1303.
https://doi.org/10.3390/math8081303

**Chicago/Turabian Style**

Leake, Carl, Hunter Johnston, and Daniele Mortari.
2020. "The Multivariate Theory of Functional Connections: Theory, Proofs, and Application in Partial Differential Equations" *Mathematics* 8, no. 8: 1303.
https://doi.org/10.3390/math8081303