A Logifold Structure for Measure Space

Abstract

In this paper, we develop a geometric formulation of datasets. The key novel idea is to formulate a dataset to be a fuzzy topological measure space as a global object and equip the space with an atlas of local charts using graphs of fuzzy linear logical functions. We call such a space a logifold. In applications, the charts are constructed by machine learning with neural network models. We implement the logifold formulation to find fuzzy domains of a dataset and to improve accuracy in data classification problems.

1. Introduction

In geometry and topology, the manifold approach dates back to Riemann using open subsets in ${\mathbb{R}}^n$ as local models to build a space. Such a local-to-global principle is central to geometry and has achieved extremely exciting breakthroughs in modeling spacetime by Einstein’s theory of relativity.

In recent years, the rapid development of data science brings immense interest to datasets that are ‘wilder’ than typical spaces that are well-studied in geometry and topology. Taming the wild is a central theme in the development of mathematics. Advances in computational tools have helped to expand the realm of mathematics in history. For instance, it took many years in human history to recognize the irrational number $\pi$ and approximate it by rational numbers. In this regard, we consider machine learning by neural network models as a modern tool to find expressions of a ‘wild space’ (for instance a dataset in real life) as a union (limit) of fuzzy geometric spaces expressed by finite formulae.

The mathematical background of this paper lies in manifold, probability, and measure theory. Only basic mathematical knowledge in these two subjects is necessary. For instance, the textbooks [1,2] provide excellent introduction to measure theory and manifold theory, respectively.

Let X be a topological space, $B_X$ the corresponding Borel $\sigma$-algebra, and $\mu$ a measure on $(X,B_X)$. We understand a dataset, for instance, the collection of all labeled appearances of cats and dogs, as a fuzzy topological measure space in nature. In the example of cats and dogs, topology concerns about the nearby appearances of the objects; measure concerns about how typical each appearance is; and fuzziness comes from how likely the objects belonging to cats, dogs, or neither.

To work with such a complicated space, we would like to have local charts that admit finite mathematical expressions and have logical interpretations, playing the role of local coordinate systems for a measure space. In our definition, a local chart $U\subset X$ is required to have a positive measure. Moreover, to avoid triviality and requiring too many charts to cover the whole space, we may further fix $ϵ>0$ and require that $\mu (X-U)<ϵ$. Such a condition disallows U to be too simple, such as a tiny ball around a point in a dataset. This resembles the Zariski-open condition in algebraic geometry.

In place of open subsets of ${\mathbb{R}}^n$, we formulate ‘local charts’ that are closely related to neural network models and have logic gate interpretations. Neural network models provide a surprisingly successful tool for finding mathematical expressions that approximate a dataset. Non-differentiable or even discontinuous functions analogous to logic gate operations are frequently used in network models. It provides an important class of non-smooth and even discontinuous functions to study a space.

We take classification problems as the main motivation in this paper. For this, we consider the graph of a function $f:D\to T$, where D is a measurable subset in ${\mathbb{R}}^n$ (with the standard Lebesgue measure) and T is a finite set (with the discrete topology). The graph $\mathrm{gr}\left(f\right)\subset D\times T$ is equipped with the push-forward measure by $D\to \mathrm{gr}\left(f\right)$.

We use the graphs of linear logical functions $f:D\to T$ explained below as local models. A chart is of the form $(U,\Phi )$, where $U\subset X$ is a measurable subset which satisfies $\mu (X-U)<ϵ$, and $\Phi :U\to \mathrm{gr}\left(f\right)$ is a measure-preserving homeomorphism. We define a linear logifold to be a pair $(X,\mathcal{U})$, where $\mathcal{U}$ is a collection of charts $(U_i,{\Phi }_i)$ such that $\mu (X-{\bigcup }_i{U}_i)=0$. In applications, the condition makes sure that the logifold covers almost every element (in the measure-theoretical sense) in the dataset. Figure 1 provides a simple example for a logifold.

In the example of cats and dogs, $U\subset X$ is a representing sample collection of appearances of cats and dogs. We take 2D images with n number of pixels for these appearances and get a function $\Phi :U\to D\times \{0,1\}$ where $T=\{0,1\}$ is the collection of the labels and $D\subset {\mathbb{R}}^n$ is the collection of pictures. The image $\Phi \left(U\right)\subset {\mathbb{R}}^n\times \{0,1\}$ is given by the graph of a function $f:D\to \{0,1\}$, which tells whether each picture shows a cat or a dog.

The definition of linear logical functions is motivated from neural network models and has a logic gate interpretation. A network model consists of a directed graph, whose arrows are equipped with linear functions and vertices are equipped with non-linear functions, which are typically ReLu or sigmoid functions in middle layers and are sigmoid or softmax functions in the last layer. The review paper [3] provides an excellent overview of deep learning models. Note that sigmoid and softmax functions are smoothings of the discrete-valued step function and the index-max function, respectively. Such smoothings are useful to describe the fuzziness of data.

From this perspective, step and index-max functions are the non-fuzzy (or called classical) limit of sigmoid and softmax functions, respectively. We will take such a limit first, and come back to fuzziness in a later stage. This means we replace all sigmoid and softmax functions in a neural network model by step and index-max functions. We will show that such a neural network is equivalent to a linear logical graph as follows: at each node of the directed graph, there is a system of N linear inequalities (on the input Euclidean domain $D\subset {\mathbb{R}}^n$) that produces $2^N$ possible Boolean outcomes for an input element, which determine the next node that this element will be passed to. We call the resulting function a linear logical function.

From the perspective of functional analysis, the functions under consideration have targets being finite sets or simplices, which are not vector spaces. Thus, the set of functions $D\to T$ is NOT a vector space. This is the main difference from the typical setting of Fourier analysis. We discuss more about this aspect using semiring structures in Section 2.5.

We prove that linear logical functions can approximate any given measurable function $f:D\to T$, where D is a measurable subset of ${\mathbb{R}}^n$ with $\mu \left(D\right)<\infty$. This provides a theoretical basis of using these functions in modeling.

Theorem 1

(Universal approximation theorem by linear logical functions). Let $f:D\to T$ be a measurable function whose domain $D\subset {\mathbb{R}}^n$ is of finite Lebesgue measure, and suppose that its target set T is finite. For any $ϵ>0$, there exists a linear logical function L and a measurable set $E\subset {\mathbb{R}}^n$ of the Lebesgue measure less than ϵ such that ${L|}_{D-E}{\equiv f|}_{D-E}$.

By taking the limit $ϵ\to 0$, the above theorem finds a linear logifold structure on the graph of a measurable function. In reality, $ϵ$ reflects the error of a network in modeling a dataset.

It turns out that for $D={\mathbb{R}}^n$, linear logical functions ${\mathbb{R}}^n\to T$ (where T is identified with a finite subset of $\mathbb{R}$) are equivalent to semilinear functions ${\mathbb{R}}^n\to T$, whose graphs are semilinear sets defined by linear equations and inequalities [4]. Semilinear sets provide the simplest class of definable sets of so-called o-minimal structures, which are closely related to model theory in mathematical logic. O-minimal structures made an axiomatic development of Grothendieck’s idea of finding tame spaces that exclude wild topology. On the one hand, definable sets have a finite expression which is crucial to the predictive power and interpretability in applications. On the other hand, our setup using measurable sets $D\subset {\mathbb{R}}^n$ provides a larger flexibility for modeling data.

Compared to more traditional approximation methods such as Fourier series, there are reasons why linear logical functions are preferred in many situations for data. When the problem is discrete in nature (for instance the target set T is finite), it is simple and natural to take the most basic kinds of discrete-valued functions as building blocks, namely step functions formed by linear inequalities. These basic functions are composed to form networks which are supported by current computational technology. Moreover, such discrete-valued functions have fuzzy and quantum deformations which have rich meanings in mathematics and physics.

Now let us address fuzziness, another important feature of a dataset besides discontinuities. In practice, there is always an ambiguity in determining whether a point belongs to a dataset. This is described as a fuzzy space $(X,\mathcal{P})$, where X a topological measure space and $\mathcal{P}:X\to (0,1]$ is a continuous measurable function that encodes the probability of whether a given point of X belongs to the fuzzy space under consideration. Here, we require $\mathcal{P}>0$ on purpose, and while points of zero probability of belonging may be adjoined to X to that the union gets simplified (for instance, X may be embedded into ${\mathbb{R}}^n$, where points in ${\mathbb{R}}^n-X$ have zero probability of belonging), they are auxiliary and have no intrinsic meaning.

Let us illustrate by the above-mentioned example of cats and dogs. The fuzzy topological measure space X consists of all possible labeled appearances of cats and dogs. The function value $\mathcal{P}\left(x\right)\in (0,1]$ expresses the probability of x belonging to the dataset, in other words, how likely the label for the appearance is correct.

To be useful, we need a finite mathematical expression (or approximation) for $\mathcal{P}$. This is where neural network models enter into the description. A neural network model for a classification problem that has the softmax function in the last layer gives a function $f=(f_1,\dots ,f_d):{\mathbb{R}}^n\to S$, where S is the standard simplex $\{{\sum }_{i=1}^d{y}_i=1\}\subset {\mathbb{R}}^d$. This gives a fuzzy space

$$({\mathbb{R}}^n\times T,\mathcal{P}),$$

where $T=\{1,\dots ,d\}$ and $\mathcal{P}(p,t):=f_t\left(p\right)$. As we have explained above, in the non-fuzzy limit, sigmoid and softmax functions are replaced by their classical counterparts of step and index-max functions, respectively, and we obtain $f^{\mathrm{classical}}:{\mathbb{R}}^n\to T$ and the subset $\{(p,t):f^{\mathrm{classical}}\left(p\right)=t\}\subset {\mathbb{R}}^n\times T$ as the classical limit. Figure 2 shows a very simple example of a fuzzy space and its classical and quantum analogs.

However, the ambient space ${\mathbb{R}}^n$ is not intrinsic; for instance, in the context of images, the dimension n gets bigger if we take images with higher resolutions, even though the objects under concern remain the same. Thus, like in manifold theory, the target space is taken to be a topological space rather than ${\mathbb{R}}^n$, and our theory takes a topological measure space X in place of ${\mathbb{R}}^n$ (or ${\mathbb{R}}^n\times T$). ${\mathbb{R}}^n\times T$ (for various possible n) is taken as an auxiliary ambient space that contains (fuzzy) measurable subsets that serve as charts to describe a dataset $(X,\mathcal{P})$.

Generally, we formulate fuzzy linear logical functions (Definition 2) and fuzzy linear logifolds (Definition 9). A fuzzy logical graph is a directed graph G whose each vertex of G is equipped with a state space, and each arrow is equipped with a continuous map between the state spaces. The walk on the graph (determined by inequalities) depends on the fuzzy propagation of the internal state spaces.

Our logifold formulation of a dataset can be understood as a geometric theory for ensemble learning and the method of Mixture of Experts. Ensemble learning utilizes multiple trained models to make a decision or prediction, see, for instance [5,6,7]. Ensemble machine learning achieves improvement in classification problems, see, for instance [8,9]. In the method of Mixture of Experts (see, for instance [10,11]), several expert models $E_j$ are employed, and there is also a gating function $G_j\left(x\right)$. The final outcome is given by the total ${\sum }_j{G}_j\left(x\right)E_j\left(x\right)$. This idea of using ‘experts’ to describe a dataset is similar to the formulation of a fuzzy logifold. On the other hand, motivated by manifold theory, we formulate universal mathematical structures that are common to datasets, namely the global intrinsic structure of a fuzzy topological measure space, and local logical structures among data points expressed by graphs of fuzzy logical functions.

The research design of this paper is characterized by its dual approach, outlined as follows: it involves rigorous mathematical constructions and formalization of logifolds and their properties, complemented by the design of algorithms, empirical validation through experiments that demonstrate their practical advantages in enhancing prediction accuracy for ensemble machine learning. This dual approach ensures both the theoretical soundness and practical utility of the proposed framework.

For readers who are more computationally oriented, they can first directly go to Section 4, where we describe the implementation of the logifold theory in algorithms. The key new ingredient in our implementation is the fuzzy domain of each model. A trained model typically does not have a perfect accuracy rate and performs well only on a subset of data points, or for a subset of target classes. A major step here is to find and record the domain of each model where it works well. Leveraging certainty scores derived from the softmax outputs of classifiers, a model’s prediction is only used in the implemented logifold structure if its certainty exceeds a predefined threshold, allowing for a refined voting system (Section 4.5).

Two experiments were conducted for logifold and they were published in our paper [12] using the refined voting system on the following well-known benchmark datasets: CIFAR10 [13], MNIST [14], and Fashion MNIST [15]. We summarize the experimental results in Section 5.

Organization

The structure of this paper is as follows. First, we formulate fuzzy linear logical functions in Section 2. Next, we establish relations with semilinear functions in Section 3.1, prove the universal approximation theorem for linear logical functions in Section 3.2, and define fuzzy linear logifolds in Section 3.3. We provide a detailed description of the algorithmic implementation of logifolds in Section 4.

2. Linear Logical Functions and Their Fuzzy Analogs

Given a subset of ${\mathbb{R}}^n$, one would like to describe it as the zero locus, the image, or the graph of a function in a certain type. In analysis, we typically think of continuous/smooth/analytic functions. However, when the domain is not open, smoothness may not be the most relevant condition.

The success of network models has taught us a new kind of function that is surprisingly powerful in describing datasets. Here, we formulate them using directed graphs and call them linear logical functions. The functions offer three distinctive advantages. First, they have the advantage of being logically interpretable in theory. Second, they are close analogues of quantum processes. Namely, they are made up of linear functions and certain non-linear activation functions, which are analogous to unitary evolution and quantum measurements. Finally, it is natural to add fuzziness to these functions and hence they are better adapted to describe statistical data.

2.1. Linear Logical Functions and Their Graphs

We consider functions $D\to T$ for $D\subset {\mathbb{R}}^n$ and a finite set T constructed from a graph as follows. Let G be a finite directed graph that has no oriented cycle and has exactly one source vertex which has no incoming arrow and $\left|T\right|$ target vertices. Each vertex that has more than one outgoing arrows is equipped with an affine linear function $l=(l_1,\dots ,l_k)$ on ${\mathbb{R}}^n$, where the outgoing arrows at this vertex are one-to-one corresponding to the chambers in ${\mathbb{R}}^n$ subdivided by the hyperplanes $\{l_i=0\}$. Explicitly, let $I\subset \{1,\dots ,k\}$, and consider

$$C_I:=\{x\in {\mathbb{R}}^n:l_i\ge 0\mathrm{if}i\in I\mathrm{and}{l}_i<0\mathrm{otherwise}\}.$$

If $C_I$ is non-empty, we call $C_I$ a chamber associated with $l=(l_1,\dots ,l_k)$.

Definition 1.

A linear logical function $f_{G,L}:D\to T$ is a function made in the following way from $(G,L)$, where G is a finite directed graph that has no oriented cycle and has exactly one source vertex and $\left|T\right|$ target vertices,

$$L=\{l_v:vs.\mathit{is}a\mathit{vertex}\mathit{with}\mathit{more}\mathit{than}\mathit{one}\mathit{outgoing}\mathit{arrows}\},$$

$l_v$ are affine linear functions whose chambers in D are one-to-one corresponding to the outgoing arrows of v. $(G,L)$ is called a linear logical graph.

Given $x\in D$, we get a path from the source vertex to one of the target vertices in G as follows. We start with the source vertex. At a vertex v, if there is only one outgoing arrow, we simply follow that arrow to reach the next vertex. If there are more than one outgoing arrows, we consider the chambers made by the affine linear function $l_v$ associated with the vertex v, and pick the outgoing arrow that corresponds to the chamber that x lies in. See Figure 3. Since the graph is finite and has no oriented cycle, we will stop at a target vertex, which is associated with an element $t\in T$. This defines the function $f_{G,L}$ by setting $f_{G,L}\left(x\right)=t$.

Proposition 1.

Consider a feed-forward network model whose activation function at each hidden layer is the step function and that at the last layer is the index-max function. The function is of the form

$$\sigma \circ L_N\circ s_{N-1}\circ L_{N-1}\circ \dots \circ s_1\circ L_1,$$

where $L_i:{\mathbb{R}}^{n_{i-1}}\to {\mathbb{R}}^{n_i}$ are affine linear functions with $n_0=n$, $s_i$ are the entrywise step functions and σ is the index-max function. We make the generic assumption that the hyperplanes defined by $L_i$ for $i=2,\dots ,N$ do not contain $s_{i-1}\circ \dots \circ L_1\left(D\right)$. Then this is a linear logical function with target $T=\{1,\dots ,n_N\}$ (on any $D\subset {\mathbb{R}}^n$, where ${\mathbb{R}}^n$ is the domain of $L_1$).

Proof.

The linear logical graph $(G,L)$ is constructed as follows. The source vertex $v_0$ is equipped with the affine linear function $L_1$. Then we make N number of outgoing arrows of $v_0$ (and corresponding vertices), where N is the number of chambers of $L_1$, which are one-to-one corresponding to the possible outcomes of $s_1$ (which form a finite subset of ${\{0,1\}}^{n_1}$). Then we consider $s_2\circ L_2$ restricted to this finite set, which also has a finite number of possible outcomes. This produces exactly one outgoing arrow for each of the vertices in the first layer. We proceed inductively. The last layer $\sigma \circ L_N$ is similar and has $n_N$ possible outcomes. Thus, we obtain $(G,L)$ as claimed, where L consists of only one affine linear function $L_1$ over the source vertex.    □

Figure 4 depicts the logical graph in the above proposition.

Proposition 2.

Consider a feed-forward network model whose activation function at each hidden layer is the ReLu function and that at the last layer is the index-max function. The function takes the form

$$\sigma \circ L_N\circ r_{N-1}\circ L_{N-1}\circ \dots \circ r_1\circ L_1,$$

where $L_i:{\mathbb{R}}^{n_{i-1}}\to {\mathbb{R}}^{n_i}$ are affine linear functions with $n_0=n$, $r_i$ are the entrywise ReLu functions, and σ is the index-max function. This is a linear logical function.

Proof.

We construct a linear logical graph $(G,L)$, which produces this function. The first step is similar to the proof of the above proposition. Namely, the source vertex $v_0$ is equipped with the affine linear function $L_1$. Next, we make N number of outgoing arrows of $v_0$ (and corresponding vertices), where N is the number of chambers of $L_1$, which are one-to-one corresponding to the possible outcomes of the sign vector of $r_1$ (which form a finite subset of ${\{0,+\}}^{n_1}$). Now we consider the next linear function $L_2$. For each of these vertices in the first layer, we consider $L_2\circ r_1\circ L_1$ restricted to the corresponding chamber, which is a linear function on the original domain $R^n$, and we equip this function to the vertex. Again, we make a number of outgoing arrows that correspond to the chambers in $R^n$ made by this linear function. We proceed inductively, and get to the layer of vertices that correspond to the chambers of $L_{N-1}\circ r_{N-2}\circ L_{N-3}\circ \dots \circ r_1\circ L_1$. Write $L_N=(l_1,\dots ,l_{n_N})$, and consider ${\tilde{L}}_N=(l_i-l_j:i\ne j)$. At each of these vertices,

$${\tilde{L}}_N\circ r_{N-1}\circ L_{N-1}\circ \dots \circ r_1\circ L_1$$

restricted on the corresponding chamber is a linear function on the original domain ${\mathbb{R}}^n$, and we equip this function to the vertex and make outgoing arrows corresponding to the chambers of the function. In each chamber, the index i that maximizes $l_i\circ r_{N-1}\circ L_{N-1}\circ \dots \circ r_1\circ L_1$ is determined, and we make one outgoing arrow from the corresponding vertex to the target vertex $i\in T$.    □

By the above propositions, ReLu/sigmoid-based feed forward network functions are linear logical functions. Thus, the above linear logical functions have the same computational complexity as the corresponding ReLu/sigmoid-based functions. Figure 5 depicts the logical graph in the above proposition.

In classification problems, T is the set of labels for elements in D, and the data determine a subset in $D\times T$ as a graph of a function. The deep learning of network models provides a way to approximate the subset as the graph of a linear logical function $\mathrm{gr}\left(f_{G,L}\right)$. Theoretically, this gives an interpretation of the dataset; namely, the linear logical graph gives a logical way to deduce the labels based on linear conditional statements on D.

The following lemma concerns the monoidal structure on the set of linear logical functions on D.

Lemma 1.

Let $f_{G_i,L_i}:D\to T_i$ be linear logical functions for $i\in I$, where $I=\{1,\dots ,k\}$. Then

$$(f_{G_i,L_i}:i\in I):D\to \prod _{i\in I}{T}_i$$

is also a linear logical function.

Proof.

We construct a linear logical graph out of $G_i$ for $i\in I$ as follows. First, take the graph $G_1$. For each target vertex of $G_1$, we equip it with the linear function at the source vertex of $G_2$ and attach to it the graph $G_2$. The target vertices of the resulting graph are labeled by $T_1\times T_2$. Similarly, each target vertex of this graph is equipped with the linear function at the source vertex of $G_3$ and attached with graph $G_3$. Inductively, we obtain the required graph, whose target vertices are labeled by ${\prod }_{i\in I}{T}_i$. By this construction, the corresponding function is $(f_{G_i,L_i}:i\in I)$.    □

$f_{(G,L)}$ admits the following algebraic expression in the form of a sum over paths which has an important interpretation in physics. The proof is straightforward and is omitted. A path in a directed graph is a finite sequence of composable arrows. The set of all linear combinations of paths and the trivial paths at vertices form an algebra by concatenation of paths.

Proposition 3.

Given a linear logical graph $(G,L)$,

$$f_{(G,L)}\left(x\right)=h\left(\sum _{\gamma }{c}_{\gamma }\left(x\right)\gamma \right)=\sum _{\gamma }{c}_{\gamma }\left(x\right)h\left(\gamma \right),$$

where the sum is over all possible paths γ in G from the source vertex to one of the target vertices; $h\left(\gamma \right)$ denotes the target vertex that γ heads to for $\gamma =a_r\dots a_1$,

$$c_{\gamma }\left(x\right)=\prod _{i=1}^r{s}_{a_i}\left(x\right),$$

where $s_a\left(x\right)=1$ if x lies in the chamber corresponding to the arrow a; otherwise, it is 0. In the above sum, exactly one of the terms is non-zero.

2.2. Zero Locus

Alternatively, we can formulate the graphs $\mathrm{gr}\left(f_{G,L}\right)$ as zero loci of linear logical functions targeted at the field ${\mathbb{F}}_2$ with two elements as follows. Such a formulation has the advantage of making the framework of algebraic geometry available in this setting.

Proposition 4.

For each linear logical function $f_{G,L}:D\to T$, there exists a linear logical function $f_{\tilde{G},\tilde{L}}:D\times T\to {\mathbb{F}}_2$ whose zero locus in $D\times T$ equals $\mathrm{gr}\left(f_{G,L}\right)$.

Proof.

Given a linear logical function $f_{G,L}:D\to T$, we construct another linear logical function $f_{\tilde{G},\tilde{L}}:D\times T\to {\mathbb{F}}_2$ as follows. Without loss of generality, let $T=\{1,\dots ,p\}$, so that $D\times T$ is embedded as a subset of ${\mathbb{R}}^{n+1}$. Any linear function on ${\mathbb{R}}^n$ is pulled back as a linear function on ${\mathbb{R}}^{n+1}$ by the standard projection ${\mathbb{R}}^{n+1}\to {\mathbb{R}}^n$ that forgets the last component. Then $f_{G,L}$ is lifted as a linear logical function $D\times T\to T$.

Consider the corresponding graph $(G,L)$. For the k-th target vertex of $(G,L)$ (that corresponds to $k\in T$), we equip it with the linear function

$$(y-(k-1/2),(k+1/2)-y):{\mathbb{R}}^{n+1}\to {\mathbb{R}}^2,$$

where y is the last coordinate of ${\mathbb{R}}^{n+1}$. This linear function produces three chambers in ${\mathbb{R}}^{n+1}$. Correspondingly, we make three outgoing arrows of the vertex. Finally, the outcome vertex that corresponds to $(+,+)$ is connected to the vertex $0\in {\mathbb{F}}_2$; the other two outcome vertices are connected to the vertex $1\in {\mathbb{F}}_2$. We obtain a linear logical graph $(\tilde{G},\tilde{L})$ and the corresponding function $f_{\tilde{G},\tilde{L}}:D\times T\to {\mathbb{F}}_2$.

By construction, $f_{\tilde{G},\tilde{L}}(x,y)=0$ for $(x,y)\in D\times T$ if and only if $y=f_{G,L}\left(x\right)\in T$. Thus, the zero locus of $f_{\tilde{G},\tilde{L}}$ is the graph of $f_{G,L}$.    □

The set of functions (with a fixed domain) valued in ${\mathbb{F}}_2$ forms a unital commutative and associative algebra over ${\mathbb{F}}_2$, which is known as a Boolean algebra.

Proposition 5.

The subset of linear logical functions $D\to {\mathbb{F}}_2$ forms a Boolean ring $\mathcal{L}$ (for a fixed $D\subset {\mathbb{R}}^n$).

Proof.

We need to show that the subset is closed under addition and multiplication induced from the corresponding operations of ${\mathbb{F}}_2$.

Let $f_{(G_1,L_1)}$ and $f_{(G_2,L_2)}$ be linear logical functions $D\to {\mathbb{F}}_2$. By Lemma 1, $(f_{(G_1,L_1)},f_{(G_2,L_2)}):D\to {\left({\mathbb{F}}_2\right)}^2$ is a linear logical function. Consider the corresponding logical graph. The target vertices are labeled by $(s_1,s_2)\in {\left({\mathbb{F}}_2\right)}^2$. We connect each of them to the vertex $s_1+s_2\in {\mathbb{F}}_2$ by an arrow. This gives a linear logical graph whose corresponding function is $f_{(G_1,L_1)}+f_{(G_2,L_2)}$. We obtain $f_{(G_1,L_1)}·f_{(G_2,L_2)}$ in a similar way.    □

In this algebro-geometric formulation, the zero locus of $f_{G,L}:D\to {\mathbb{F}}_2$ corresponds to the ideal $\left(f_{G,L}\right)\subset \mathcal{L}$.

2.3. Parameterization

The graph $\mathrm{gr}\left(f_{G,L}\right)$ of a linear logical function can also be put in parametric form. For the moment, we assume the domain D is finite. First, we need the following lemma.

Lemma 2.

Assume $D\subset {\mathbb{R}}^n$ is finite. Then the identity function $I_D:D\to D$ is a linear logical function.

Proof.

Since D is finite, there exists a linear function $l:{\mathbb{R}}^n\to {\mathbb{R}}^N$ such that each chamber of l contains at most one point of D. Then we construct the linear logical graph G as follows. The source vertex is equipped with the linear function l and outgoing arrows corresponding to the chambers of l. Elements of D are identified as the target vertices of these arrows which correspond to chambers that contain them. The corresponding function $f_{G,L}$ equals $I_D$.    □

Proposition 6.

Given a linear logical function $f_{G,L}:D\to T$ with finite $\left|D\right|$, there exists an injective linear logical function $D\to D\times T$ whose image equals $\mathrm{gr}\left(f_{G,L}\right)$.

Proof.

By Lemmas 1 and 2, $(I_D,f_{G,L})$ is a linear logical function. By definition, its image equals $\mathrm{gr}\left(f_{G,L}\right)$.    □

2.4. Fuzzy Linear Logical Functions

Another important feature of a dataset is its fuzziness. Below, we formulate the notion of a fuzzy linear logical function and consider its graph. Basic notions of fuzzy logic can be found in textbooks such as [16]. There are many developed applications of fuzzy logic such as modeling, control, pattern recognition and networks, see, for instance [17,18,19,20,21].

Definition 2.

Let G be a finite directed graph that has no oriented cycle and has exactly one source vertex and target vertices $t_1,\dots ,t_K$ as in Definition 1. Each vertex v of G is equipped with a product of standard simplices

$$P_v=\prod _{k=1}^{m_v}{S}^{d_{v,k}},S^{d_{v,k}}=\left\{(y_1,\dots ,y_{d_{v,k}})\in {\mathbb{R}}_{\ge 0}^{d_{v,k}}:\sum _{i=1}^{d_{v,k}}{y}_i\le 1\right\}$$

for some integers $m_v>0$, $d_{v,k}\ge 0$. $P_v$ is called the internal state space of the vertex v. Let D be a subset of the internal state space of the source vertex of G. Each vertex v that has more than one outgoing arrow is equipped with an affine linear function

$$l_v:\prod _{k=1}^{m_v}{\mathbb{R}}^{d_{v,k}}\to {\mathbb{R}}^j$$

for some $j>0$, and we require that the collection of non-empty intersections $C_I\cap P_v$ of the chambers $C_I$ of $l_v$ with the product simplex $P_v$ are one-to-one corresponding to the outgoing arrows of v. Let L denote the collection of these affine linear functions $l_v$. Moreover, each arrow a is equipped with a continuous function

$$p_a:P_{s\left(a\right)}\to P_{t\left(a\right)},$$

where $s\left(a\right),t\left(a\right)$ denote the source and target vertices, respectively.

We call $(G,L,P,p)$ a fuzzy linear logical graph. Let $P^{\mathit{out}}$ denote the disjoint union ${\coprod }_{l=1}^K{P}_{t_l}$. $(G,L,P,p)$ determines a function

$$f_{(G,L,P,p)}:D\to P^{\mathit{out}}$$

as follows. Given $x\in D$, it is an element of the internal state space $P_v$ of the source vertex v. Moreover, it lies in a unique chamber of the affine linear function $l_v$ at the source vertex v. Let $v^{\prime }$ be the head vertex of the outgoing arrow a corresponding to this chamber. We have the element $p_a\left(x\right)\in P_{v^{\prime }}$. By repeating this process, we obtain a path from the source vertex to one of the target vertices $t_l$, and also an element in the internal state space $P_{t_l}$ that we define as $f_{(G,L,P,p)}\left(x\right)$. The resulting function $f_{(G,L,P,p)}$ is called a fuzzy linear logical function.

In the above definition, a standard choice of a continuous map between product simplices

$$\prod _{j=1}^m{S}^{d_j}\to \prod _{k=1}^n{S}^{r_k}$$

is using the softmax function $\tilde{\sigma }$:

$${\left(\tilde{\sigma }\circ l_k({\overrightarrow{x}}_1,\dots ,{\overrightarrow{x}}_m)\right)}_{k=1}^n,$$

where $l_k:{\prod }_{j=1}^m{\mathbb{R}}^{d_j}\to {\mathbb{R}}^{r_k}$ are affine linear functions. $\tilde{\sigma }:{\mathbb{R}}^r\to S^r$ is defined by

$$\tilde{\sigma }(z_1,\dots ,z_r)=\left({\displaystyle \frac{e^{z_1}}{1+{\sum }_{i=1}^r{e}^{z_i}}},\dots ,{\displaystyle \frac{e^{z_r}}{1+{\sum }_{i=1}^r{e}^{z_i}}}\right).$$

This is the sigmoid function when $r=1$.

Remark 1.

We can regard the definition of a fuzzy linear function as a generalization of the linear logical function in Definition 1. Note that the linear functions $l_v$ in L in the above definition have domain to be the internal state spaces over the corresponding vertices v. In comparison, the linear functions $l_v$ in L in Definition 1 have domain as the input space ${\mathbb{R}}^n$. A linear logical graph $(G,L)$ in Definition 1 has no internal state space except the ${\mathbb{R}}^n$ at the input vertex.

To relate the two notions given by the above definition and Definition 1, we can set $P_v=S^n$ as the same for all vertices v except the target vertices $t_1,\dots ,t_K$, which are equipped with the zero-dimensional simplex (a point), and we set $p_a$ as the identity maps for all arrows a that are not targeted at any of $t_i$. Then $f_{(G,L,P,p)}$ reduces back to a linear logical function in Definition 1.

Remark 2.

We call the corners of the convex set $P_v$ state vertices, which take the form $e_I=(e_{i_1},\dots ,e_{i_{m_v}})\in P_v$ for a multi-index $I=(i_1,\dots ,i_{m_v})$, where $\{e_0,\dots ,e_{d_{v,k}}\}\subset {\mathbb{R}}^{d_{v,k}+1}$ is the standard basis. We have a bigger graph $\widehat{G}$ by replacing each vertex v of G by the collection of state vertices over v and each arrow of G by the collection of all possible arrows from source state vertices to target state vertices. Then the vertices of G are interpreted as ‘layers’ or ‘clusters’ of vertices of $\widehat{G}$. The input state $x\in D$, the arrow linear functions L, and the maps between state spaces p determine the probability of getting to each target state vertex of $\widehat{G}$ from the source vertex.

In this interpretation, we take the target set to be the disjoint union of corners of $P_{t_l}$ at the target vertices $t_1,\dots ,t_K$ as follows:

$$T=\coprod _{l=1}^K{T}_l:=\coprod _{l=1}^K\left\{e_I:I=(i_1,\dots ,i_{m_{t_l}})\mathrm{for}{i}_k\in \{0,\dots ,d_{t_l,k}\}\right\},$$

(1)

which is a finite set. The function $f=f_{(G,L,P,p)}$ determines the probability of the outcome for each input state $x\in D$ as follows. Let $f\left(x\right)\in P_{t_l}={\prod }_{k=1}^{m_{t_l}}{S}^{d_{t_l,k}}$ for some $l=1,\dots ,K$. Then the probability of being in $T_j$ is zero for $j\ne l$. Writing $f\left(x\right)=(f_1\left(x\right),\dots ,f_{m_{t_l}}\left(x\right))$ for $f_k\left(x\right)\in S^{d_{t_l,k}}$, the probability of the output to be $t=e_I\in T_l$ for $I=(i_1,\dots ,i_{m_{t_l}})$ is given by ${\prod }_{k=1}^{m_{t_l}}{f}_k^{\left(i_k\right)}\left(x\right)\in [0,1]$.

Proposition 7.

Consider the function

$$f=\tilde{\sigma }\circ L_N\circ {\tilde{s}}_{N-1}\circ L_{N-1}\circ \dots \circ {\tilde{s}}_1\circ L_1$$

given by a feed-forward network model whose activation function at each hidden layer is the sigmoid function (denoted by ${\tilde{s}}_i$), and that at the last layer it is the softmax function $\tilde{\sigma }$. f is a fuzzy linear logical function.

Proof.

We set $(G,L,P,p)$ as follows. G is the graph that has $(N+1)$ vertices $v_0,\dots ,v_N$ with arrows $a_i$ from $v_{i-1}$ to $v_i$ for $i=1,\dots ,N$. L is just an empty set. $P_i:={\left(S^1\right)}^{m_i}$ for $i=0,\dots ,N-1$, where $m_i$ is the dimension of the domain of $L_{i+1}$. The one-dimensional simplex $S^1$ is identified with the interval $[0,1]$. $P_N:=S^{m_N}$, where $m_N$ is the dimension of the target of $L_N$. Then

$$\begin{array}{c}\hfill p_i:={\tilde{s}}_i\circ L_i{|}_{{[0,1]}^{m_{i-1}}}\mathrm{for}i=1,\dots ,N-1\\ \hfill p_N:=\tilde{\sigma }\circ L_N{|}_{{[0,1]}^{m_{N-1}}}.\end{array}$$

Then $f=f_{(G,L,P,p)}$.    □

Proposition 8.

Consider the function

$$f=\tilde{\sigma }\circ L_N\circ r_{N-1}\circ L_{N-1}\circ \dots \circ r_1\circ L_1$$

given by a feed-forward network model whose activation function at each hidden layer is the ReLu function (denoted by $r_i$), and that at the last layer it is the softmax function $\tilde{\sigma }$. Here, each $L_i$ denotes an affine linear function. The function f is a fuzzy linear logical function.

Proof.

We need to construct a fuzzy linear logical graph $(G,L,P,p)$ such that $f=f_{(G,L,P,p)}$. We take G to be the logical graph constructed in Example 2 (Figure 5) with the last two layers of vertices replaced by a single target vertex t. Each vertex that points to the last vertex t and t itself only has zero or one outgoing arrow and hence is not equipped with a linear function. Other vertices are equipped with linear functions on the input space ${\mathbb{R}}^n$ as in Example 2. We take the internal state space to be the n-dimensional cube $P_v:={\left(S^1\right)}^n\subset {\mathbb{R}}^n$, where n is the input dimension at every vertex v except at the target vertex, whose internal state space is defined to be the simplex $P_t:=S^d$, where d is the target dimension of f. The function $p_a$ in Definition 2 is defined to be the identity function on the internal state space $P_v$ for every arrow a except for the arrows that point to t. Now, we need to define $p_a$ for the arrows that point to t. Let $t_i^{\prime }$ be the source vertices of these arrows. The input space ${\mathbb{R}}^n$ is subdivided into chambers $\{x\in {\mathbb{R}}^n:\mathrm{the}\mathrm{path}\mathrm{determined}\mathrm{by}x\mathrm{targets}\mathrm{at}{t}_i^{\prime }\}$. Moreover, $L_N\circ r_{N-1}\circ L_{N-1}\circ \dots \circ r_1\circ L_1$ is a piecewise-linear function, whose restrictions on each of these chambers is linear and extend to a linear function l on ${\mathbb{R}}^n$. Then $p_a$ for the corresponding arrow a is defined to be $\tilde{\sigma }\circ l$. By this construction, we have $f=f_{(G,L,P,p)}$.    □

As in Proposition 3, $f_{(G,L,P,p)}$ can be expressed in the form of the sum over paths.

Proposition 9.

Given a fuzzy linear logical graph $(G,L,P,p)$,

$$f_{(G,L,P,p)}\left(x\right)=\sum _{\gamma }{c}_{\gamma }\left(x\right)p_{\gamma }\left(x\right),$$

where the sum is over all possible paths γ in G from the source vertex to one of the target vertices; for $\gamma =a_r\dots a_1$, $p_{\gamma }\left(x\right)={\prod }_{i=1}^r{p}_{a_r}\dots p_{a_1}\left(x\right)$;

$$c_{\gamma }\left(x\right)=\prod _{i=1}^r{s}_{a_i}\left(p_{a_{i-1}\dots a_1}\left(x\right)\right),$$

where $s_a\left(x\right)=1$ if $x\in P_{t_a}$ lies in the chamber corresponding to the arrow a or is 0 otherwise. In the above sum, exactly one of the terms is non-zero.

2.5. The Semiring of Fuzzy Linear Logical Functions

Since the target space $P^{out}$ is not a vector space, the set of functions $D\to P^{out}$ does not form a vector space. This makes a main difference from usual approximation theory (such as Fourier analysis or Taylor series) where the function space is flat as the target is a vector space (typically $\mathbb{R}$). For a fuzzy logic target (the product simplex in our case), the ‘function space’ forms a semiring. Due to this difference, we shall work out some well-known results in functional analysis in the semiring context below.

Recall that a semi-group is a set equipped with a binary operation that is associative; a monoid is a semi-group with an identity element. A semiring is a set equipped with two binary operations addition and multiplication which satisfy the ring axioms except for the existence of additive inverses. In particular, a semiring is a semi-group under addition and a semi-group under multiplication, and the two operations are compatible in the sense that multiplication distributes over addition. A semiring without a unit is a semiring that does not have a multiplicative identity.

Example 1

(Viterbi semiring). Let $S^1=[0,1]$ be the unit interval with the usual multiplication and addition defined as taking maximum. The additive identity is 0 and the multiplicative identity is 1. Then $(S^1,+,·)$ is a semiring. This is known as the Viterbi semiring that appears in probabilistic parsing.

The subset of two elements $\{0,1\}\subset [0,1]$ is a sub-semiring known as the Boolean semiring. Addition (taking maximum) and multiplication are identified as the logical operations OR and AND, respectively.

In this subsection, we focus on the case that the target space $P^{out}$ is a product of intervals ${\prod }_{k=1}^m{S}^1={[0,1]}^m$.

Lemma 3.

The set of functions from a set D to ${[0,1]}^m$ (or ${\{0,1\}}^m$) has a semiring structure.

Proof.

For the semiring structure on ${[0,1]}^m$ (or ${\{0,1\}}^m$), we simply take entrywise addition and multiplication, where addition is defined as taking maximum and multiplication is the usual one. The additive identity is the origin $\overrightarrow{0}$ and the multiplicative identity is $(1,\dots ,1)$. The set of functions to ${[0,1]}^m$ (or ${\{0,1\}}^m$) has a semiring structure by taking maximum and multiplication of function values. The additive identity is the zero function that takes all domain points to $\overrightarrow{0}$.    □

Remark 3.

If we take a disjoint union of product simplices in place of ${[0,1]}^m$, we still have a semi-group structure by entriwise multiplication. However, since taking the entriwise maximum does not preserve the inequality ${\sum }_i{x}_i\le 1$ for a simplex, this destroys the semiring structure. This is why we restrict ourselves to ${[0,1]}^m$ in this section.

Theorem 2.

Fix D as a subset of a product simplex, and fix the target product of intervals $P^{out}={[0,1]}^n$. The set of fuzzy linear logical functions $f:D\to P^{out}$ is a sub-semiring of the semiring of functions $D\to P^{out}$.

Similarly, the set of linear logical functions $f:D\to {\{0,1\}}^n$ is a sub-semiring of the semiring of functions $D\to {\{0,1\}}^n$.

Proof.

Given two linear logical functions $f_{(G_1,L_1,P_1,p_1)}$ and $f_{(G_2,L_2,P_2,p_2)}$ with the same domain $D\subset P^{in}$ and target $P^{out}$, we need to show that their sum (in the sense of entriwise maximum) and product are also linear logical functions. We construct a directed graph G with one source vertex and one target vertex by merging the target vertex of $G_1$ with the source vertex of $G_2$ as a single vertex, and then we add one outgoing arrow to the final vertex of $G_2$ to another vertex, which is the target of G.

Now for the internal state spaces, over those vertices v of G that belong to $G_1$, we take ${\left(P_1\right)}_v\times P^{in}$; over those v that belong to $G_2$, we take $P^{out}\times {\left(P_2\right)}_v$. Note that the vertex that comes from merging of the target vertex of $G_1$ and the source vertex of $G_2$ is equipped with the internal state space $P^{out}\times P^{in}$; the vertex that corresponds to the final vertex of $G_2$ is $P^{out}\times P^{out}$. The last vertex is equipped with the space $P^{out}$.

The arrow maps are defined as follows. For arrows a that come from $G_1$, we take the function $p_1\left(a\right)\times \mathrm{Id}$. For arrows a that come from $G_2$, we take the function $\mathrm{Id}\times p_2\left(a\right)$. By this construction for an input $x\in D$, the value becomes $(f_{(G_1,L_1,P_1,p_1)}\left(x\right),f_{(G_2,L_2,P_2,p_2)}\left(x\right))$ at the vertex corresponding to the final vertex of $G_2$. The last arrow is then equipped with the continuous function $P^{out}\times P^{out}\to P^{out}$ given by the entriwise maximum (or entriwise product). Thus, $(G,L,P,p)$ gives the function $f_{(G,L,P,p)}$, which is the sum (or the product) of $f_{(G_1,L_1,P_1,p_1)}$ and $f_{(G_2,L_2,P_2,p_2)}$.

The proof for the case of linear logical functions $f:D\to {\{0,1\}}^n$ is similar and hence omitted.    □

$P^{out}={[0,1]}^n$ is equipped with the standard Euclidean metric $d_{P^{out}}$. Moreover, the domain D is equipped with the Lebesgue measure $\mu$. We consider the set of measurable functions $D\to P^{out}$.

Lemma 4.

The set of measurable functions $D\to P^{out}$ form a sub-semiring.

Proof.

If $f,g:D\to P^{out}$ are measurable functions, then $(f,g):D\to P^{out}\times P^{out}$ is also measurable. Taking entriwise maximum and multiplication are continuous functions $P^{out}\times P^{out}\to P^{out}$. Thus, the compositions, which give the sum and product of f and g, are also measurable functions. Furthermore, the zero function is measurable. Thus, the set of measurable functions $D\to P^{out}$ forms a sum semiring.    □

We define

$$d(f,g):={\int }_D{d}_{P^{out}}(f\left(x\right),g\left(x\right))d\mu \left(x\right)={\int }_D\parallel f\left(x\right)-g\left(x\right)\parallel d\mu \left(x\right).$$

To make this a distance function such that $d(f,g)=0$ implies $f=g$, we need to quotient out the functions whose support has measure zero.

Proposition 10.

The subset of measurable functions $f:D\to P^{out}$ whose support $\{f\ne \overrightarrow{0}\}$ has measure zero forms an ideal of the semiring of measurable functions $D\to P^{out}$. The above function d gives a metric on the corresponding quotient semiring, denoted by $L^0(D,P^{out})$.

Proof.

The support of the sum $max(f,g)$ is the union of the supports of f and g. The support of the product $f·g$ is the intersection of the supports of f and g. Thus, if the supports of f and g have measure zero, then so does the support of $max(f,g)$. Furthermore, if f has the support of measure zero, $f·g$ has the support of measure zero (with no condition on g). This shows that the subset of measurable functions $f:D\to P^{out}$ with the support of measure zero forms an ideal.

d is well-defined on the quotient semiring because if f and g are in the same equivalent class, that is $f+{\delta }_1=g+{\delta }_2$ for some measurable functions ${\delta }_1,{\delta }_2$ whose support have measure zero, then ${f|}_{D-A}{=g|}_{D-A}$ for some set A of measure zero. Thus, for any other function h,

$$\begin{array}{cc}\hfill d(f,h)=& {\int }_D{d}_{P^{out}}(f\left(x\right),h\left(x\right))d\mu \left(x\right)={\int }_{D-A}{d}_{P^{out}}(f\left(x\right),h\left(x\right))d\mu \left(x\right)\hfill \\ \hfill =& {\int }_{D-A}{d}_{P^{out}}(g\left(x\right),h\left(x\right))d\mu \left(x\right)={\int }_D{d}_{P^{out}}(g\left(x\right),h\left(x\right))d\mu \left(x\right)=d(g,h).\hfill \end{array}$$

Now, consider any two elements $\left[f\right],\left[g\right]$ in the quotient semiring. It is obvious that $d\left(\right[f],[g\left]\right)=d\left(\right[g],[f\left]\right)$ from the definition of d. Furthermore, the following triangle inequality holds:

$$\begin{array}{cc}\hfill d\left(\right[f],[g\left]\right)=& {\int }_D\parallel f\left(x\right)-g\left(x\right)\parallel d\mu \left(x\right)\hfill \\ \hfill \le & {\int }_D\parallel f\left(x\right)-h\left(x\right)\parallel d\mu \left(x\right)+{\int }_D\parallel h\left(x\right)-g\left(x\right)\parallel d\mu \left(x\right)\hfill \\ \hfill =& d\left(\right[f],[h\left]\right)+d\left(\right[h],[g\left]\right).\hfill \end{array}$$

Finally, suppose $d\left(\right[f],[g\left]\right)=0$. This means that ${\int }_D\parallel f\left(x\right)-g\left(x\right)\parallel d\mu \left(x\right)=0$. Since the integrand is non-negative, we have $\parallel f\left(x\right)-g\left(x\right)\parallel =0$ for almost every $x\in D$. Thus, $f\left(x\right)=g\left(x\right)$ away from a measure zero subset $A\subset D$. Since $f\left(x\right)=1_{D-A}f+1_{A}f$ and $g\left(x\right)=1_{D-A}g+1_{A}g$ (where $1_A$ denotes the characteristic function of the set A which takes value 1 on A and 0 otherwise), we have

$$\left[f\right]=\left[1_{D-A}f\right]=\left[1_{D-A}g\right]=\left[g\right].$$

   □

Proposition 11.

$L^0(D,P^{out})$ is a topological semiring (under the topology induced from the metric d).

Proof.

We need to show that the addition and multiplication operations

$$max(f,g),f·g:L^0(D,P^{out})\times L^0(D,P^{out})\to L^0(D,P^{out})$$

is continuous. First, consider

$$(f,g):L^0(D,P^{out})\times L^0(D,P^{out})\to L^0(D,P^{out}\times P^{out}).$$

This is continuous, as follows: for $ϵ>0$, consider $\tilde{f},\tilde{g}$ with $d(\tilde{f},f)<\delta$ and $d(\tilde{g},g)<\delta$. Then

$$\begin{array}{cc}\hfill d((\tilde{f},\tilde{g}),(f,g))=& {\int }_D{d}_{P^{out}\times P^{out}}((\tilde{f}\left(x\right),\tilde{g}\left(x\right)),(f\left(x\right),g\left(x\right)))d\mu \left(x\right)\hfill \\ \hfill & \le {\int }_D|\tilde{f}-f|d\mu +{\int }_D|\tilde{g}-g|d\mu <2\delta =ϵ\hfill \end{array}$$

by taking $\delta =ϵ/2$. Now, the function $max(a,b):P^{out}\times P^{out}\to P^{out}$ is Lipschitz continuous with Lipschitz constant 1. Entriwise multiplication $P^{out}\times P^{out}\to P^{out}$ is also Lipschitz continuous since the product $P^{out}\times P^{out}$ is compact. Denote either one of these two functions by $\varphi$, and let K be the Lipschitz constant. As above, we have chosen suitable $\delta$ to control $\tilde{f}$ and $\tilde{g}$ such that

$${\int }_D{d}_{P^{out}\times P^{out}}((\tilde{f}\left(x\right),\tilde{g}\left(x\right)),(f\left(x\right),g\left(x\right)))d\mu \left(x\right)<ϵ.$$

Then

$$\begin{array}{cc}\hfill d(\varphi \circ (\tilde{f},\tilde{g}),\varphi \circ (f,g))=& {\int }_D{d}_{P^{out}}(\varphi (\tilde{f}\left(x\right),\tilde{g}\left(x\right)),\varphi (f\left(x\right),g\left(x\right)))d\mu \left(x\right)\hfill \\ \hfill & \le K{\int }_D{d}_{P^{out}\times P^{out}}((\tilde{f}\left(x\right),\tilde{g}\left(x\right)),(f\left(x\right),g\left(x\right)))d\mu \left(x\right)<Kϵ.\hfill \end{array}$$

   □

2.6. Graph of Fuzzy Linear Logical Function as a Fuzzy Subset

A fuzzy subset of a topological measure space X is a continuous measurable function $\mathcal{F}:X\to [0,1]$. This generalizes the characteristic function of a subset. The interval $[0,1]$ can be equipped with the semiring structure whose addition and multiplication are taking maximum and minimum, respectively. This induces a semiring structure on the collection of fuzzy subsets $\mathcal{F}:X\to [0,1]$ that plays the role of union and intersection operations.

The graph $\mathrm{gr}\left(f\right)$ of a function

$$f:D\to \coprod _{l=1}^K{P}_{t_l},$$

where $P_{t_l}$ are products of simplices as in Definition 2, is defined to be the fuzzy subset in $D\times T$, where T is defined by Equation (1), given by the probability at every $(x,t)\in D\times T$ determined by f in Remark 2.

The following is a fuzzy analog of Proposition 4.

Proposition 12.

Let $f_{(G,L,P,p)}$ be a fuzzy linear logical function. Let $\mathcal{F}:D\times T\to [0,1]$ be the characteristic function of its graph where T is defined by (1). Then $\mathcal{F}$ is also a fuzzy linear logical function.

Proof.

Similar to the proof of Proposition 4, we embed the finite set T as the subset $\{1,\dots ,|T\left|\right\}\subset \mathbb{R}$. The affine linear functions on ${\mathbb{R}}^n\supset D$ in the collection L are pulled back as affine linear functions on ${\mathbb{R}}^{n+1}\supset D\times T$. Similarly for the input vertex $v_0$, we replace the product simplex $P_{v_0}$ by ${\tilde{P}}_{v_0}:=P_{v_0}\times [0,|T|+1]$ (where the interval $[0,|T|+1]$ is identified with $S^1$); for the arrows a tailing at $v_0$, $p_a$ are pulled back to be functions ${\tilde{P}}_{v_0}\to P_{h\left(a\right)}$. Then, we obtain $(\tilde{L},\tilde{P},\tilde{p})$ on G.

For each of the target vertices $t_l$ of G, we equip it with the linear function

$$(y-3/2,y-5/2,\dots ,y-(\left|T\right|-1/2\left)\right):{\mathbb{R}}^{n+1}\to {\mathbb{R}}^{\left|T\right|-1},$$

where y is the last coordinate of the domain ${\mathbb{R}}^{n+1}$. It divides ${\mathbb{R}}^{n+1}$ into $\left|T\right|$ chambers that contain ${\mathbb{R}}^n\times \left\{j\right\}$ for some $j=1,\dots ,\left|T\right|$. Correspondingly, we make $\left|T\right|$ outgoing arrows of the vertex $t_l$. The new vertices are equipped with the internal state space $P_{t_l}$, and the new arrows are equipped with the identity function $P_{t_l}\to P_{t_l}$. Then we get $\left|T\right|K$ additional vertices, where K is the number of target vertices $t_l$ of G. Let us label these vertices as $v_{j,l}$ for $j\in T$ and $l\in \{1,\dots ,K\}$. Each of these vertices are connected to the new output vertex by a new arrow. The new output vertex is equipped with the internal state space $S^1\cong [0,1]$. The arrow from $v_{j,l}$ to the output vertex is equipped with the following function $p_{j,l}:P_{v_{j,l}}\to [0,1]$. If $j\notin T_l$, then we set $p_{j,l}\equiv 0$. Otherwise, for $j=e_I\in T_l$ and $I=(i_1,\dots ,i_{m_{t_l}})$, $p_{j,l}:={\prod }_{k=1}^{m_{t_l}}{u}_k^{\left(i_k\right)}$, where $u_k^{\left(0\right)},\dots ,u_k^{\left(d_{t_l,k}\right)}$ are the coordinates of $S^{d_{t_l,k}}\subset {\mathbb{R}}_{\ge 0}^{d_{t_l,k}+1}$. This gives the fuzzy linear logical graph $(\tilde{G},\tilde{L},\tilde{P},\tilde{p})$ whose associated function $f_{(\tilde{G},\tilde{L},\tilde{P},\tilde{p})}:D\times T\to [0,1]$ is the characteristic function.    □

The above motivates us to consider fuzzy subsets whose characteristic functions are fuzzy linear logical functions $\mathcal{F}:X\to [0,1]$. Below, we show that they form a sub-semiring, that is, they are closed under fuzzy union and intersection. We need the following lemma analogous to Lemma 1.

Lemma 5.

Let $f_{G_i,L_i,P_i,p_i}:D\to P_i^{\mathit{out}}$ be fuzzy linear logical functions for $i\in I$, where $I=\{1,\dots ,k\}$, and we assume that the input state space $P_{i,\mathrm{in}}$ are the same for all i. Then

$$(f_{G_i,L_i,P_i,p_i}:i\in I):D\to \prod _{i\in I}{P}_i^{\mathrm{out}}$$

is also a fuzzy linear logical function.

Proof.

By the proof of Lemma 1, we obtain a new graph $(G,L)$ from $(G_i,L_i)$ for $i=1,\dots ,k$ by attaching $(G_{i+1},L_{i+1})$ to the target vertices of $(G_i,L_i)$. For the internal state spaces, we change as follows. First, we make a new input vertex ${\tilde{v}}_0$ and an arrow ${\tilde{a}}_0$ from ${\tilde{v}}_0$ to the original input vertex $v_0$ of $(G,L)$. We denote the resulting graph by $(\tilde{G},\tilde{L})$. We define ${\tilde{P}}_{{\tilde{v}}_0}:=P_{v_0}$, ${\tilde{P}}_{v_0}:={\prod }_{i=1}^k{P}_{v_0}$, where $P_{v_0}=P_{i,\mathrm{in}}$ for all i by assumption, and ${\tilde{p}}_{{\tilde{a}}_0}:P_{v_0}\to {\prod }_{i=1}^k{P}_{v_0}$ to be the diagonal map ${\tilde{p}}_{{\tilde{a}}_0}=(\mathrm{Id},\dots ,\mathrm{Id})$. The internal state spaces $P_v$ over vertices v of $G_1$ are replaced by $P_v\times {\prod }_{i=2}^k{P}_{v_0}$, and $p_a$ for arrows a of $G_1$ are replaced by $(p_a,\mathrm{Id},\dots ,\mathrm{Id})$. Next, over the vertices v of the graph $G_2$ that is attached to the target vertex $t_l$ of $G_1$, the internal state space $P_v$ is replaced by $P_{t_l}\times P_v\times {\prod }_{i=3}^k{P}_{v_0}$, and $p_a$ for arrows a of $G_2$ are replaced by $(\mathrm{Id},p_a,\mathrm{Id},\dots ,\mathrm{Id})$. Inductively, we obtain the desired graph $(\tilde{G},\tilde{L},\tilde{P},\tilde{p})$.    □

Proposition 13.

Suppose ${\mathcal{F}}_1,{\mathcal{F}}_2:X\to [0,1]$ are fuzzy subsets defined by fuzzy linear logical functions. Then ${\mathcal{F}}_1\cup {\mathcal{F}}_2$ and ${\mathcal{F}}_1\cap {\mathcal{F}}_2$ are also fuzzy subsets defined by fuzzy linear logical functions.

Proof.

By the previous lemma, $({\mathcal{F}}_1,{\mathcal{F}}_2)=f_{(G,L,P,p)}:X\to [0,1]\times [0,1]$ for some fuzzy linear logical graph $(G,L,P,p)$, which has a single output vertex whose internal state space is $[0,1]\times [0,1]\cong S^1\times S^1$. We attach an arrow a to this output vertex. Over the new target vertex v, $P_v:=[0,1]$; $p_a:=max:[0,1]\times [0,1]\to [0,1]$ (or $p_a:=min$). Then, we obtain $(\tilde{G},\tilde{L},\tilde{P},\tilde{p})$, whose corresponding fuzzy function defines ${\mathcal{F}}_1\cup {\mathcal{F}}_2$ (or ${\mathcal{F}}_1\cap {\mathcal{F}}_2$, respectively).    □

Remark 4.

For $f:P^{\mathit{in}}\to P^{\mathit{out}}$, where $P^{\mathit{in}},P^{\mathit{out}}$ are product simplices, we can have various interpretations.

1.

As a usual function, its graph is in the product $P^{\mathit{in}}\times P^{\mathit{out}}$.

2.

As a fuzzy function on $P^{\mathit{in}}$: $P^{\mathit{in}}\to T$, where T is the finite set of vertices of the product simplex $P^{\mathit{out}}$, its graph is a fuzzy subset in $P^{\mathit{in}}\times T$.

3.

The domain product simplex $P^{\mathit{in}}$ can also be understood as a collection of fuzzy points over V, the finite set of vertices of $P^{\mathit{in}}$, where a fuzzy point here just refers to a probability distribution (which integrates to 1).

4.

Similarly, $P^{\mathit{in}}\times P^{\mathit{out}}$ can be understood as a collection of fuzzy points over $V\times T$. Thus, the (usual) graph of f can be interpreted as a sub-collection of fuzzy points over $V\times T$.

$(\mathrm{Id},f)$ gives a parametric description of the graph of a function f. The following ensures that it is a fuzzy linear logical function if f is.

Corollary 1.

Let $f:D\to P^{\mathrm{out}}$ be a fuzzy linear logical function. Then $(\mathrm{Id},f):D\to D\times P^{\mathrm{out}}$ is also a fuzzy linear logical function whose image is the graph of f.

Proof.

By Lemma 5, it suffices to know that $\mathrm{Id}:D\to D$ is a fuzzy linear logical function. This is obvious: we take the graph with two vertices serving as input and output, which are connected by one arrow. The input and output vertices are equipped with the internal state spaces that contain D, and p is just defined by the identity function.    □

Remark 5.

Generative deep learning models that are widely used nowadays can be understood as parametric descriptions of data sets X by fuzzy linear logical functions $f:D\to X$ (where D and X are embedded in certain product simplices $P^{\mathrm{in}}$ and $P^{\mathrm{out}}$, respectively, and D is usually called the noise space). We focus on classification problems in the current work and plan to extend the framework to other problems as well in the future.

2.7. A Digression to Non-Linearity in a Quantum-Classical System

The fuzzy linear logical functions in Definition 2 have the following quantum analog. Quantum systems and quantum random walks are well known and studied, see, for instance [22,23]. On the other hand, they depend only linearly on the initial state in probability. The motivation of this subsection is to compare fuzzy and quantum systems and to show how non-linear dependence on the initial probability distribution can come up. On the other hand, this section is mostly unrelated to the rest of the writing and can be skipped.

Definition 3.

Let G be a finite directed graph that has no oriented cycle and has exactly one source vertex and target vertices $t_1,\dots ,t_K$. Each vertex v is equipped with a product of projectifications of Hilbert spaces over complex numbers as follows:

$$Q_v:=\prod _{l=1}^{m_v}\mathbb{P}\left(H_{v,l}\right)$$

for some integer $m_v>0$. We fix an orthonormal basis in each Hilbert space $H_{v,l}$, which gives a basis in the tensor product as follows:

$$E^{\left(v\right)}=\left\{e_I^{\left(v\right)}=(e_{i_1}^{\left(v\right)},\dots ,e_{i_{m_v}}^{\left(v\right)}):e_{i_l}^{\left(v\right)}\mathit{is}\mathit{a}\mathit{basic}\mathit{vector}\mathit{of}{H}_{v,l}\right\}.$$

For each vertex v that has more than one outgoing arrows, we make the choice of decomposing set $E^{\left(v\right)}$ into subsets that are one-to-one corresponding to the outgoing arrows. Each arrow a is equipped with a map $q_a$ from the corresponding subset of basic vectors $e_I^{\left(t\right(a\left)\right)}$ to $Q_{h\left(a\right)}$.

Let us call the tuple $(G,Q,E,q)$ a quantum logical graph.

We obtain a probabilistic map $f_{(G,Q,E,q)}:Q^{\mathrm{in}}\to T$ as follows. Given a state $\overrightarrow{w}=(w_1,\dots ,w_{m_v})\in Q_v$ at a vertex v, we make a quantum measurement and $\overrightarrow{w}$ projects to one of the basic elements $e_I^{\left(v\right)}$ with probability ${\prod }_{l=1}^{m_v}{\left|\langle w_l,e_{i_l}^{\left(v\right)}\rangle \right|}^2$. The outcome $e_I^{\left(v\right)}$ determines which outgoing arrow a to pick, and the corresponding map $q_a$ sends it to an element of $Q_{h\left(a\right)}$. Inductively, we obtain an element $f_{(G,Q,E,q)}\left(w\right)\in T$.

However, such a process is simply linearly depending on the initial condition in probability as follows: the probabilities of outcomes of the quantum process $f_{(G,Q,E,q)}\left(w\right)$ for an input state w (which is complex-valued) simply linearly depends on the modulus of components of the input w. In other words, the output probabilities are simply obtained by a matrix multiplication on the input probabilities. To produce non-linear physical phenomena, we need the following extra ingredient.

Let us consider the state space ${\mathbb{P}}^n$ of a single particle. A basis gives a map $\mu :{\mathbb{P}}^n\to S^n$ to the simplex $S^n$ (also known as the moment map of a corresponding torus action on ${\mathbb{P}}^n$):

$$\mu =\left({\displaystyle \frac{|z_0{|}^2}{|z_0{|}^2+\dots +{\left|z_n\right|}^2}},\dots ,{\displaystyle \frac{|z_n{|}^2}{|z_0{|}^2+\dots +{\left|z_n\right|}^2}}\right):{\mathbb{P}}^n\to S^n.$$

The components of the moment map are the probability of quantum projection to basic states of the particle upon observation. By the law of large numbers, if we make independent observations of particles in an identical quantum state $\overrightarrow{z}=[z_0:\dots :z_n]\in {\mathbb{P}}^n$ for N times, the average of the observed results (which are elements in $\left\{\right(1,0,\dots ,0),\dots ,(0,\dots ,0,1\left)\right\}$) converges to $\mu \left(\overrightarrow{z}\right)\in S^n$ as $N\to \infty$.

The additional ingredient we need is the choice of a map $s:S^n\to {\mathbb{P}}^m$ and $m,n\in {\mathbb{Z}}_{>0}$. For instance, when $m=n=1$, we set the initial phase of the electron spin state according to a number in $[0,1]$. Upon an observation of a state, we obtain a point in $\{(1,0,\dots ,0),\dots ,(0,\dots ,0,1)\}\subset S^n$. Now, if we have N particles simultaneously observed, we obtain N values, whose average is again a point p in the simplex $S^n$. By s, these are turned to N quantum particles in state $s\left(p\right)\in {\mathbb{P}}^m$ again.

$\mu :{\mathbb{P}}^n\to S^n$ and $s:S^n\to {\mathbb{P}}^m$ give an interplay between quantum processes and classical processes with averaging. Averaging in the classical world is the main ingredient to produce non-linearity from the linear quantum process.

Now, let us modify Definition 3 by using $s:S^n\to {\mathbb{P}}^m$. Let $P_v$ be the product simplex corresponding to $Q_v$ at each vertex. Moreover, as in Definitions 1 and 2 for (fuzzy) linear logical functions, we equip each vertex with affine linear functions $l_v$ whose corresponding systems of inequalities divide $P_v$ into chambers. This decomposition of $P_v$ plays the role of the decomposition of $E^{\left(v\right)}$ in Definition 3. The outgoing arrows at v are in a one-to-one correspondence with the chambers. Each outgoing arrow a at v is equipped with a map ${\tilde{q}}_a$ from the corresponding chamber of $P_v$ to $Q_{h\left(a\right)}$. ${\tilde{q}}_a$ can be understood as an extension of $q_a$ (whose domain is a subset of corners of $P_{t\left(a\right)}$) in Definition 3.

Definition 4.

We call the tuple $(G,Q,E,L,\tilde{q})$ (where L is the collection of affine linear functions $l_v$) a quantum-classical logical graph.

Given N copies of the same state $\overrightarrow{w}$ in $Q_v$, we first take a quantum projection of these and they become elements in $P_v$. We take an average of these N elements, which lies in a certain chamber defined by $l_v$. The chamber corresponds to an outgoing arrow a, and the map ${\tilde{q}}_a$ produces N elements in $Q_{h\left(a\right)}$. Inductively, we obtain a quantum-classical process $Q^{\mathrm{in}}\to T$.

For $N=1$, this essentially produces the same linear probabilistic outcomes as in Definition 3. On the other hand, when $N>1$, the process is no longer linear and produces a fuzzy linear logical function $P^{\mathrm{in}}\to P^{\mathrm{out}}$.

In summary, non-linear dependence on the initial state results from averaging of observed states.

Remark 6.

We can allow loops or cycles in the above definition. Then the system may run without stop. In this situation, the main object of concern is the resulting (possibly infinite) sequence of pairs $(v,s)$, where v is a vertex of G and s is a state in $Q_v$. This gives a quantum-classical walk on the graph G.

We can make a similar generalization for (fuzzy) linear logical functions by allowing loops or cycles. This is typical in applications in time-dependent network models.

3. Linear Logical Structures for a Measure Space

In the previous section, we have defined linear logical functions based on a directed graph. In this section, we first show the equivalence between our definition of linear logical functions and semilinear functions [4] in the literature. Thus, the linear logical graph we have defined can be understood as a representation of semilinear functions. Moreover, fuzzy and quantum logical functions that we define can be understood as deformations of semilinear functions.

Next, we consider measurable functions and show that they can be approximated and covered by semilinear functions. This motivates the definition of a logifold, which is a measure space that has graphs of linear logical functions as local models.

3.1. Equivalence with Semilinear Functions

Let us first recall the definition of semilinear sets.

Definition 5.

For any positive integer n, semilinear sets are the subsets of ${\mathbb{R}}^n$ that are finite unions of sets of the form

$$\{x\in {\mathbb{R}}^n:f_1\left(x\right)=\dots =f_k\left(x\right)=0,g_1\left(x\right)>0,\dots ,g_l\left(x\right)>0\},$$

(2)

where the $f_i$ and $g_j$ are affine linear functions.

A function $f:D\to T$ on $D\subset {\mathbb{R}}^n$, where T is a discrete set, is called semilinear if for every $t\in T$, $f^{-1}\left\{t\right\}$ equals to the intersection of D with a semilinear set.

Now let us consider linear logical functions defined in the last section. We show that the two notions are equivalent (when the target set is finite). Thus, a linear logical graph can be understood as a graphical representation (which is not unique) of a semilinear function. From this perspective, the last section provides fuzzy and quantum deformations of semilinear functions.

Theorem 3.

Consider $f:D\to T$ for a finite set $T=\{t_1,\dots ,t_s\}$, where $D\subset {\mathbb{R}}^n$. f is a semilinear function if and only if it is a linear logical function.

Proof.

It suffices to consider the case $D={\mathbb{R}}^n$. We use the following terminologies for convenience. Let $V\left(G\right)$ and $E\left(G\right)$ be the sets of vertices and arrows, respectively, for a directed graph G. A vertex $v\in V\left(G\right)$ is called nontrivial if it has more than one outgoing arrows. It is said to be simple if it has exactly one outgoing arrow. We call a vertex that has no outgoing arrow a target and that which has no incoming arrow a source. For a target t, let $R_t$ be the set of all paths from the source to target t.

Consider a linear logical function $f=f_{(G,L)}:{\mathbb{R}}^n\to T$. Let p be a path in $R_t$ for some $t\in T$. Let $\{v_1,\dots ,v_k\}$ be the set of non-trivial vertices that p passes through. This is a non-empty set unless f is just a constant function (recall that G has only one source vertex). At each of these vertices $v_i$, ${\mathbb{R}}^n$ is subdivided according to the affine linear functions $g_{i,1},\dots ,g_{i,N_i}$ into chambers $C_{i,1},\dots ,C_{i,m_i}$, where $m_i$ is the number of its outgoing arrows. All the chambers $C_{i,j}$ are semilinear sets.

For each path $p\in R_t$, we define a set $E_p$ such that $x\in E_p$ if x follows path p to get the target t. Then $E_p$ can be represented as $C_{1,j_1}\cap \dots \cap C_{k,j_k},$ which is semilinear. Moreover, the finite union

$$f^{-1}\left(t\right)=\bigcup _{p\in R_t}{E}_p$$

is also a semilinear set. This shows that f is a semilinear function.

Conversely, suppose that we are given a semilinear function. Without loss of generality, we can assume that f is surjective. For every $t\in T$, $f^{-1}\left(t_i\right)$ is a semilinear set defined by a collection of affine linear functions in the form of (2). Let $\mathcal{F}=\{l_1,\dots ,l_N\}$ be the union of these collections over all $t\in T$.

Now, we construct a linear logical graph associated with f. We consider the chambers made by $(l_1,-l_1,\dots ,l_N,-l_N)$ by taking the intersection of the half spaces $l_i\ge 0$, $l_i<0$, $-l_i\ge 0$, $-l_i<0$. We construct outgoing arrows of the source vertex associated with these chambers.

For each $t\in T$, $l_j$ occurs in defining $f^{-1}\left(t\right)$ as either one of the following ways:

• $l_j>0$, which is equivalent to $l_j\ge 0$ and $-l_j<0$,
• $l_j=0$, which is equivalent to $l_j\ge 0$ and $-l_j\ge 0$,
• $l_j$ is not involved in defining $f^{-1}\left(t\right)$.

Thus, $f^{-1}\left(t\right)$ is a union of a sub-collection of chambers associated with the outgoing arrows. Then, we assign these outgoing arrows with the target vertex t. This is well-defined since $f^{-1}\left(t\right)$ for different t are disjoint to each other. Moreover, since ${\bigcup }_t{f}^{-1}\left(t\right)={\mathbb{R}}^n$, every outgoing arrow is associated with a certain target vertex.

In summary, we have constructed a linear logical graph G which produces the function f.    □

The above equivalence between semilinear functions and linear logical functions naturally generalizes to definable functions in other types of o-minimal structures. They provide the simplest class of examples in o-minimal structures for semi-algebraic and subanalytic geometry [4]. The topology of sub-level sets of definable functions was recently investigated in [24]. Let us first recall the basic definitions.

Definition 6

 ([4]). A structure $\mathcal{S}$ on $\mathbb{R}$ consists of a Boolean algebra ${\mathcal{S}}_n$ of subsets of ${\mathbb{R}}^n$ for each $n=0,1,2,\dots ,$ such that

1.

the diagonals $\{x\in {\mathbb{R}}^n:x_i=x_j\},1\le i<j\le n$ belong to ${\mathcal{S}}_n$;

2.

$A\in {\mathcal{S}}_m,B\in {\mathcal{S}}_n\Rightarrow A\times B\in {\mathcal{S}}_{m+n}$;

3.

$A\in {\mathcal{S}}_{n+1}\Rightarrow \pi \left(A\right)\in {\mathcal{S}}_n$, where $\pi :{\mathbb{R}}^{n+1}\to {\mathbb{R}}^n$ is the projection map defined by $\pi (x_1,\dots ,x_{n+1})=(x_1,\dots ,x_n)$;

4.

the ordering $\{(x,y)\in {\mathbb{R}}^2:x<y\}$ of $\mathbb{R}$ belongs to ${\mathcal{S}}_2$.

A structure $\mathcal{S}$ is o-minimal if the sets in ${\mathcal{S}}_1$ are exactly the subsets of $\mathbb{R}$ that have only finitely many connected components, that is, the finite unions of intervals and points.

Given a collection $\mathcal{A}$ of subsets of the Cartesian spaces ${\mathbb{R}}^n$ for various n, such that the ordering $\left\{\right(x,y):x<y\}$ belongs to $\mathcal{A}$, define $Def\left(\mathcal{A}\right)$ as the smallest structure on the real line containing $\mathcal{A}$ by adding the diagonals to $\mathcal{A}$ and closing off under Boolean operations, Cartesian products, and projections. Sets in $Def\left(\mathcal{A}\right)$ are said to be definable from $\mathcal{A}$ or simply definable if $\mathcal{A}$ is clear from context.

Given definable sets $A\subset {\mathbb{R}}^m$ and $B\subset {\mathbb{R}}^n$ we say that a map $f:A\to B$ is definable if its graph $\mathrm{\Gamma }\left(f\right)=\{(x,f\left(x\right))\in {\mathbb{R}}^{m+n}:x\in A\}$ is definable.

Remark 7.

If $\mathcal{A}$ consists of the ordering, the singletons $\left\{r\right\}$ for any $r\in \mathbb{R}$, the graph in ${\mathbb{R}}^2$ of scalar multiplications maps $x\mapsto \lambda x:\mathbb{R}\to \mathbb{R}$ for any $\lambda \in \mathbb{R}$, and the graph of addition $\{(x,y,z)\in {\mathbb{R}}^3:z=x+y\}$. Then $Def\left(\mathcal{A}\right)$ consists of semilinear sets for various positive integers n (Definition 5).

Similarly, if $\mathcal{A}$ consist of the ordering, singletons, and the graphs of addition and multiplication, then $Def\left(\mathcal{A}\right)$ consists of semi-algebraic sets, which are finite unions of sets of the form

$$\left\{x\in {\mathbb{R}}^n:f\left(x\right)=0,g_1\left(x\right)>0,\dots ,g_l\left(x\right)>0\right\},$$

where f and $g_1,\dots ,g_l$ are real polynomials in n variables, due to the Tarski–Seidenberg Theorem [25].

One obtains semi-analytic sets in which the above $f,g_1,\dots ,g_l$ become real analytic functions by including graphs of analytic functions. Let an be the collection $\mathcal{A}$ and of the functions $f:{\mathbb{R}}^n\to \mathbb{R}$ for all positive integers n such that ${f|}_{I^n}$ is analytic, $I=[-1,1]\subset \mathbb{R}$, and f is identically 0 outside the cubes. The theory of semi-analytic sets and subanalytic sets show that $Def\left(\mathit{an}\right)$ is o-minimal, and relatively compact semi-analytic sets have only finitely many connected components. See [25] for efficient exposition of the ojasiewicz-Gabrielov-Hironaka theory of semi- and subanalytic sets.

Theorem 4.

Let us replace the collection of affine linear functions at vertices in Definition 1 by polynomials and call the resulting functions polynomial logical functions. Then $f:D\to T$ for a finite set T is a semi-algebraic function if and only if f is a polynomial logical function.

The proof of the above theorem is similar to that of Theorem 3 and hence omitted.

3.2. Approximation of Measurable Functions by Linear Logical Functions

We consider measurable functions $f:D\to T$, where $0<\mu \left(D\right)<\infty$ and T is a finite set. The following approximation theorem for measurable functions has two distinct features since T is a finite set. First, the functions under consideration, and linear logical functions that we use, are discontinuous. Second, the ‘approximating function’ actually exactly equals to the target function in a large part of D. Compared to traditional approximation methods, linear logical functions have an advantage of being representable by logical graphs, which have fuzzy or quantum generalizations.

Theorem 5

(Universal approximation theorem for measurable functions). Let μ be the standard Lebesgue measure on ${\mathbb{R}}^n$. Let $f:D\to T$ be a measurable function with $\mu \left(D\right)<\infty$ and a finite target set T. For any $ϵ>0$, there exists a linear logical function $L:D\to \tilde{T}$, where $\tilde{T}=T\cup \{\ast \}$ is T adjunct with a singleton, and a measurable set $E\subset D$ with $\mu \left(E\right)<ϵ$ such that ${L|}_{D-E}{\equiv f|}_{D-E}$.

Proof.

Let $\mathcal{R}:=\left\{{\prod }_{k=1}^n(a_k,b_k]\subset {\mathbb{R}}^n:a_k<b_k\mathrm{for}\mathrm{all}\mathrm{k}\right\}$ be the family of rectangles in ${\mathbb{R}}^n$. We use the well-known fact that for any measurable set U of finite Lebesgue measure, there exists a finite subcollection $\left\{R_j:j=1,\dots ,N\right\}$ of $\mathcal{R}$ such that $\mu \left(U▵{\bigcup }_1^N{R}_j\right)<ϵ$ (see, for instance [1]). Here, $A▵B:=(A-B)\cup (B-A)$ denotes the symmetric difference of two subsets $A,B$.

Suppose that a measurable function $f:D\to T=\{t_1,\dots ,t_m\}$ and $ϵ>0$ be given. For each $t\in T$, let ${\mathcal{S}}_t$ be a union of finitely many rectangles of $\mathcal{R}$ that approximates $f^{-1}\left(t\right)\subset D$ (that has finite measure) in the sense that $\mu \left((f^{-1}\left(t\right)▵{\mathcal{S}}_t)\right)<{\displaystyle \frac{ϵ}{\left|T\right|}}$. Note that ${\mathcal{S}}_t$ is a semilinear set.

The case $m=1$ is trivial. Suppose $m>1$. Define semilinear sets ${\mathcal{S}}_{*}=\bigcup _{i<j}\left({\mathcal{S}}_{t_i}\cap {\mathcal{S}}_{t_j}\right)$ and ${\mathfrak{S}}_t={\mathcal{S}}_t\setminus {\mathcal{S}}_{*}$ for each $t\in T$. Now, we define $L:D\to \tilde{T}$,

$$L\left(p\right)=\left\{\begin{array}{cc}t\hfill & \mathrm{if}p\in {\mathfrak{S}}_t\cap D\hfill \\ \ast \hfill & \mathrm{if}p\in D\setminus \bigcup _{t\in T}{\mathfrak{S}}_t\hfill \end{array}\right.$$

which is a semilinear function on D.

If $p\in {\mathcal{S}}_{*}\cap D$, then $p\in {\mathcal{S}}_{t_i}\cap {\mathcal{S}}_{t_j}$ for some $t_i,t_j\in T$ with $t_i\ne t_j$. In such a case, $p\in f^{-1}\left(t_i\right)$ implies $p\notin f^{-1}\left(t_j\right)$. It shows ${\mathcal{S}}_{*}\cap D\subset \bigcup _{t\in T}\left({\mathcal{S}}_t\setminus f^{-1}\left(t\right)\right)$. Furthermore, we have $D\setminus \bigcup _{t\in T}{\mathcal{S}}_t\subset \bigcup _{t\in T}\left(f^{-1}\left(t\right)\setminus {\mathcal{S}}_t\right)$. Therefore

$$D\setminus \bigcup _{t\in T}{\mathfrak{S}}_t=\left(D\setminus \bigcup _{t\in T}{\mathcal{S}}_t\right)\cup \left(D\cap {\mathcal{S}}_{*}\right)\subset \bigcup _{t\in T}{f}^{-1}\left(t\right)▵{\mathcal{S}}_t,$$

and hence

$$\mu \left(D\setminus \bigcup _{t\in T}{\mathfrak{S}}_t\right)\le \sum _{t\in T}\mu \left(f^{-1}\left(t\right)▵{\mathcal{S}}_t\right)<ϵ.$$

By Theorem 3, L is a linear logical function.    □

Corollary 2.

Let $f:D\to T$ be a measurable function where $D\subset {\mathbb{R}}^n$ is of finite measure and T is finite. Then there exists a family $\mathcal{L}$ of linear logical functions $L_i:D_i\to T$, where $D_i\subset D$ and $L_i{\equiv f|}_{D_i}$, such that $D\setminus \bigcup _i{D}_i$ is a measure zero set.

3.3. Linear Logifold

To be more flexible, we can work with the Hausdorff measure which is recalled as follows.

Definition 7.

Let $p\ge 0$, $\delta >0$. For any $U\subset {\mathbb{R}}^n$, $Diam\left(U\right)$ denotes the diameter of U defined by the supremum of distance of any two points in U. For a subset $E\subset {\mathbb{R}}^n$, define

$$H_{\delta }^p\left(E\right):=\underset{{\mathcal{U}}_{\delta }\left(E\right)}{inf}\sum _{U\in {\mathcal{U}}_{\delta }\left(E\right)}Diam{\left(U\right)}^p,$$

where ${\mathcal{U}}_{\delta }\left(E\right)$ denotes a cover of E by sets U with $Diam\left(U\right)<\delta$. Then the p-dimensional Hausdorff measure is defined as $H^p\left(E\right):=\underset{\delta \to 0}{lim}{H}_{\delta }^p\left(E\right)$. The Hausdorff dimension of E is ${dim}_H\left(E\right):={\displaystyle \underset{p}{inf}}\{p\in [0,\infty ):H^p\left(E\right)=0\}$.

Definition 8.

A linear logifold is a pair $(X,\mathcal{U})$, where X is a topological space equipped with a σ-algebra and a measure μ, $\mathcal{U}$ is a collection of pairs $(U_i,{\varphi }_i)$, where $U_i$ are subsets of X such that $\mu \left(U_i\right)>0$ and $\mu (X-{\bigcup }_i{U}_i)=0$; ${\varphi }_i$ are measure-preserving homeomorphisms between $U_i$ and the graphs of linear logical functions $f_i:D_i\to T_i$ (with an induced Hausdorff measure), where $D_i\subset {\mathbb{R}}^{n_i}$ are $H^{p_i}$-measurable subsets in certain dimension $p_i$, and $T_i$ are discrete sets.

The elements of $\mathcal{U}$ are called charts. A chart $(U,\varphi )$ is called entire up to measure ϵ if $\mu (X-U)<ϵ$.

Comparing to a topological manifold, we require $\mu \left(U_i\right)>0$ in place of an openness condition. Local models are now taken to be graphs of linear logical functions in place of open subsets of Euclidean spaces.

Then, the results in the last subsection can be rephrased as follows.

Corollary 3.

Let $f:D\to T$ be a measurable function on a measurable set $D\subset {\mathbb{R}}^n$ of finite measure with a finite target set T. For any $ϵ>0$, its graph $\mathrm{gr}\left(f\right)\subset D\times T$ can be equipped with a linear logifold structure that has an entire chart up to measure ϵ.

Remark 8.

In [26], relations between neural networks and quiver representations were studied. In [27,28], a network model is formulated as a framed quiver representation; learning of the model was formulated as a stochastic gradient descent over the corresponding moduli’s space. In this language, we now take several quivers, and we glue their representations together (in a non-linear way) to form a ‘logifold’.

In a similar manner, we define a fuzzy linear logifold below. By Remark 2 and (2) of Remark 4, a fuzzy linear logical function has a graph as a fuzzy subset of $D\times T$. We are going to use the fuzzy graph as a local model for a fuzzy space $(X,\mathcal{P})$.

Definition 9.

A fuzzy linear logifold is a tuple $(X,\mathcal{P},\mathcal{U})$, where

1.

X is a topological space equipped with a measure μ;.

2.

$\mathcal{P}:X\to (0,1]$ is a continuous measurable function.

3.

$\mathcal{U}$ is a collection of tuples $({\rho }_i,{\varphi }_i,f_i)$, where ${\rho }_i$ are measurable functions ${\rho }_i:X\to [0,1]$ with ${\sum }_i{\rho }_i\le 1_X$ that describe fuzzy subsets of X, whose supports are denoted by by $U_i=\{x\in X:{\rho }_i\left(x\right)>0\}\subset X$;

$${\varphi }_i:U_i\to D_i\times T_i$$

are measure-preserving homeomorphisms where $T_i$ are finite sets in the form of (1) and $D_i\subset {\mathbb{R}}^{n_i}$ are $H^{p_i}$-measurable subsets in certain dimension $p_i$; $f_i$ are fuzzy linear logical functions on $D_i$ whose target sets are $T_i$, as described in Remark 2.

4.

the induced fuzzy graphs ${\mathcal{F}}_i:D_i\times T_i\to [0,1]$ of $f_i$ satisfy

$$\mathcal{P}=\sum _i{\rho }_i·{\varphi }_i^{*}\left({\mathcal{F}}_i\right).$$

(3)

Persistent homology [29,30,31] can be defined for fuzzy spaces $(X,\mathcal{P})$ by using the filtration $X_c:=\{x\in X:\mathcal{P}\left(x\right)\ge c\}\subset X$ for $c\in [0,1]$ associated with $\mathcal{P}$. We plan to study persistent homology for fuzzy logifolds in a future work.

4. Ensemble Learning and Logifolds

In this section, we briefly review the ensemble learning methods in [32] and make a mathematical formulation via logifolds. Moreover, in Section 4.4 and Section 4.5, we introduce the concept of fuzzy domain and develop a refined voting method based on this. We view each trained model as a chart given by a fuzzy linear logical function. The domain of each model can be a proper subset of its feature space defined by the inverse image of a proper subset of the target classes. For each trained model, a fuzzy domain is defined using the certainty score for each input, and only inputs which lie in this certain part are accepted. In [12], we demonstrated in experiments that this method produces improvements in accuracy compared to taking the average of outputs.

4.1. Mathematical Description of Neural Network Learning

Consider a subset $\mathcal{Z}$ of ${\mathbb{R}}^n\times T$, where $T=\{c_1,\dots ,c_N\}$, which we take as the domain of a model. ${\mathbb{R}}^n$ is typically referred to as the feature space, while each $c_i\in T$ represents a class. We embed T as the corners of the standard simplex $S^{N-1}\subset {\mathbb{R}}^N$. One wants to find an expression of the probability distribution of $\mathcal{Z}$ in terms of a function produced by a neural network.

Definition 10.

The underlying graph of a neural network is a finite directed graph G. Each vertex v is associated with ${\mathbb{R}}^{n_v}$ for some $n_v\in {\mathbb{Z}}_{>0}$, together with a non-linear function ${\mathbb{R}}^{n_v}\to {\mathbb{R}}^{n_v}$ called an activation function.

Let Θ be the vector space of linear representations. A linear representation associates each arrow $\overrightarrow{a}$ with a linear map ${\mathbb{R}}^{n_{s\left(\overrightarrow{a}\right)}}\to {\mathbb{R}}^{n_{t\left(\overrightarrow{a}\right)}}$, where $s\left(\overrightarrow{a}\right),t\left(\overrightarrow{a}\right)$ are the source and target vertices, respectively.

Let us fix γ to be a linear combination of paths between two fixed vertices s and t in G. The associated network function $f_{\theta }:{\mathbb{R}}^{n_s}\to {\mathbb{R}}^{n_t}$ for each $\theta \in \Theta$ is defined to be the corresponding function obtained by the sum of compositions of linear functions and activation functions along the paths of γ.

One would like to minimize the function $C_{\mathcal{Z}}:\Theta \to \mathbb{R}$,

$$C_{\mathcal{Z}}\left(\theta \right):=\sum _{(x,y)\in \mathcal{Z}}{\parallel f_{\theta }\left(x\right)-y\parallel }^2,$$

which measures the distance between the graph of $f_{\theta }$ and $\mathcal{Z}$. To do this, one takes a stochastic gradient descent(SGD) over $\Theta$. In a discrete setting, it is given by the following equation:

$${\theta }_{k+1}={\theta }_k-\eta \nabla C_{\mathcal{Z}}-\eta W_k,$$

where $\eta \in R_{>0}$ is called the step size or learning rate, $W_k$ is the $k^{\mathrm{th}}$noise or Brownian Motion, and $\nabla C_{\mathcal{Z}}$ denotes the gradient vector field of $C_{\mathcal{Z}}$ (in practice, the sample $\mathcal{Z}$ is divided into batches and C is the sum over a batch).

For practical purposes, the completion of the computational process is marked by the verification of epochs. Then the hyper-parameter space $\mathcal{H}$ for SGD is a subspace $\left\{(\eta ,\mathrm{Batch}\mathrm{size},\mathrm{Epochs},\mathrm{Noise})\right\}\subset {\mathbb{R}}^3\times \mathcal{D}$, where $\mathcal{D}$ is the space of $\mathbb{R}$-valued random variables with zero mean and finite variance. This process is called the training procedure, and the resulting function $g=f_{{\theta }_{*}}$ is called a trained model, where ${\theta }_{*}$ is the minimizer. The $argmaxg\left(x\right)$ is called the prediction of g at x, which is well-defined almost everywhere. For $c_i\in T$, $g(x,c_i):=g{\left(x\right)}_i$ is called the certainty of the model for x being in class $c_i$.

4.2. A Brief Description of Ensemble Machine Learning

Ensemble machine learning utilizes more than one classifiers to make decisions. Dasarathy and Sheela [33] were early contributors to this theory, who proposed the partitioning feature space using multiple classifiers. Ensemble systems offer several advantages, including smoothing decision boundaries, reducing classifier bias, and addressing issues related to data volume. A widely accepted key to a successful ensemble system is achieving diversity among its classifiers. Refs. [7,34,35] provide good reviews of this theory.

Broadly speaking, designing an ensemble system involves determining how to obtain classifiers with diversity and how to combine their predictions effectively. Here, we briefly introduce popular methods. Bagging [36], short for bootstrap aggregating, trains multiple classifiers, each on a randomly sampled subset of the training dataset. Boosting, such as AdaBoost(Adaptive Boosting) [37,38] iteratively trains classifiers by focusing on the instances they misclassified in previous rounds. In Mixture of Experts [11], each classifier specializes in different tasks or subsets of dataset, with a ‘gating’ layer which determines weights for the combination of classifiers.

Given multiple classifiers, an ensemble system makes decisions based on predictions from diverse classifiers and a rule for combining predictions is necessary. This is usually done by taking a weighted sum of the predictions, see, for instance [39]. Moreover, the weights may also be tuned via a training process.

4.3. Logifold Structure

Let $(X,\mathcal{P})$ be a fuzzy topological measure space with $\mu \left(X\right)>0$, where $\mu$ is the measure of X, which is taken as an idealistic dataset. For instance, it can be the set of all possible appearances of cats, dogs, and eggs. A sample fuzzy subset U of X is taken and is identified with a subset of ${\mathbb{R}}^n\times T$. This identification is denoted by $\varphi :U\to {\mathbb{R}}^n\times T$ and $\mathcal{Z}:=\varphi \left(U\right)$. For instance, this can be obtained by taking pictures in a certain number of pixels for some cats and dogs, and T is taken as the subset of labels {‘C’,‘D’}. By the mathematical procedure given above, we obtain a trained model g, which is a fuzzy linear logical function, denoted by $g=g_{(G,L,P,p)}:{\mathbb{R}}^n\to S^{\left|T\right|-1}$, where G is the neural network with one target vertex, and L and p are the affine linear maps and activation functions, respectively. This is the concept of a chart of X in Definition 9.

Let $g_i:{\mathbb{R}}^{n_i}\to S^{|T_i|-1}$ be a number of trained models and $G_i:{\mathbb{R}}^{n_i}\times T_i\to [0,1]$ be the corresponding certainty functions. Note that $n_i$ and $T_i$ can be distinct for different models. Their results are combined according to certain weight functions ${\rho }_i:X\to [0,1]$ and we get

$$\sum _i{\rho }_i\left(x\right)G_i\left({\varphi }_i\left(x\right)\right):X\to [0,1],$$

where the sum is over those i whose corresponding charts $U_i$ contain x.

This gives $\mathcal{P}$ in (3), and we obtain a fuzzy linear logifold.

In the following sections, we introduce the implementation details for finding fuzzy domains and the corresponding voting system for models with different domains.

4.4. Thick Targets and Specialization

We consider fuzzy subsets $U_i\subset X$ and ${\mathcal{Z}}_i\subset {\mathbb{R}}^{n_i}\times T_i$ with identification ${\varphi }_i:U_i\to {\mathcal{Z}}_i$. A common fuzziness that we make use of comes from ‘thick targets’. For instance, let $T=\{‘C’,‘D’,‘E’\}$ (continuing the example used in the last subsection). Consider $\tilde{T}=\{\{‘C’,‘D’\},\{‘E’\}\}$, which consists of the two classes $\{‘C’,‘D’\}$ and $\{‘E’\}$. We take a sample $\tilde{\mathcal{Z}}\subset {\mathbb{R}}^n\times \tilde{T}$ consisting of pictures of cats, dogs, and eggs with the two possible labels ‘cats or dogs’ and ‘eggs’. Then, we train a model with two targets ($|\tilde{T}|=2$), and obtain $\tilde{G}:{\mathbb{R}}^n\times \tilde{T}\to [0,1]$.

Definition 11.

Let T be a finite set and $\tilde{T}$ be a subset of the power set $\mathcal{P}\left(T\right)$ such that $\varnothing \notin \tilde{T}$ and $t\cap t^{\prime }=\varnothing$ for all distinct $t,t^{\prime }\in \tilde{T}$.

1.

$t\in \tilde{T}$ is called thin (or fine) if it is a singleton and thick otherwise.

2.

The union $\bigcup \tilde{T}\subset T$ is called the flattening of $\tilde{T}$.

3.

We say that $\tilde{T}$ is full if its flattening is T and fine if all its elements are thin.

Given a model $g_i:{\mathbb{R}}^{n_i}\to S^{|{\tilde{T}}_i|-1}$, where $g_i=(g_{i,1},\dots ,g_{i,|{\tilde{T}}_i|})$, define the certainty function of $g_i$ as $C_i:={max}_j{g}_{i,j}$. For $\alpha \in [0,1]$,

$${\mathcal{Z}}_{\alpha ,i}:=\{(x,y)\in {\mathcal{Z}}_i:C_i\left(x\right)\ge \alpha \}$$

is called the certain part of the model $g_i$, or fuzzy domain of $g_i$, at certainty threshold $\alpha$ in ${\mathcal{Z}}_i$. Let $\mathcal{M}$ denote the collection of trained model ${\left\{g_i\right\}}_{i\in \mathcal{I}}$ with identifications ${\varphi }_i:U_i\subset X\to {\mathcal{Z}}_i$. The union $X_{\alpha }={\bigcup }_{i\in \mathcal{I}}{\varphi }_i^{-1}\left({\mathcal{Z}}_{\alpha ,i}\right)$ is called the certain part with certainty threshold α of $\mathcal{M}$. For instance, in the dataset of appearances of dogs, cats, and eggs, suppose we have a model $g_i$ with target $T_i=\{‘C’,‘D’\}$. ${\varphi }_i^{-1}\left({\mathcal{Z}}_{\alpha ,i}\right)$ is the subset of appearance of cats and dogs sampled by the set of labeled pictures ${\mathcal{Z}}_i\subset {\mathbb{R}}^{n_i}\times T_i$ that has certainty $\ge \alpha$ by the model. Note that as $\alpha$ decreases, there must be a greater or equal number of models satisfying the conditions, in particular, $X_{\alpha }\subset X_{{\alpha }^{\prime }}$ for ${\alpha }^{\prime }<\alpha$.

Table 1. Summary of frequently used symbols. Let X denote the dataset equipped with a topology and a measure representing its distribution in real-world space. Following Definition 9, we define the fuzzy linear Logifold as the collection $\mathcal{M}={\left\{(U_i,{\varphi }_i,g_i)\right\}}_{i\in \mathcal{I}}$, where each triple consists of a domain $U_i\subset X$, an identification map ${\varphi }_i:U_i\to {\mathbb{R}}^{n_i}\times {\tilde{T}}_i$, and a trained model $g_i$. For brevity, we refer to $\mathcal{M}$ as the set of trained models ${\left\{g_i\right\}}_{i\in \mathcal{I}}$. Note that $U_i={\mathcal{Z}}_{\alpha ,i.}$ for some certainty threshold $\alpha$ in application.

Symbol	Meaning
X	A fuzzy topological measure space (dataset)
$\mathcal{M}$	A collection of trained models ${\left\{g_i\right\}}_{i\in \mathcal{I}}$
$(U_i,{\varphi }_i,g_i)$	The identification ${\varphi }_i:U_i\subset X\to {\mathbb{R}}^{n_i}\times {\tilde{T}}_i$,
	${\mathcal{Z}}_i:={\varphi }_i\left(U_i\right)$ and ${\tilde{T}}_i$ the target of $g_i$.
$T_i:=\bigcup {\tilde{T}}_i$	Flattening of targets of the i-th model
$g_{i,j}:=j-\mathrm{th}\mathrm{component}\mathrm{of}{g}_i$	The certainty of $g_i$ to $t_j\in {\tilde{T}}_i$
$C_i:={max}_j{g}_{i,j},$	The certainty function of $g_i$
$P_i:=t_{arg{max}_j{g}_{i,j}}$	The prediction function of $g_i$
${\mathcal{Z}}_{\alpha ,i}\subset {\mathbb{R}}^{n_i}\times T_i$	The certain part by $g_i$
	at certainty threshold $\alpha$ in ${\mathcal{Z}}_i$
$X_{\alpha }={\bigcup }_{i\in \mathcal{I}}{\varphi }_i^{-1}\left({\mathcal{Z}}_{\alpha ,i}\right)\subset X$	The certain part of X
	at certainty threshold $\alpha$ by $\mathcal{M}$

summarizes the notations introduced here.

One effective method we use to generate more charts of different types is to change the target of a model, which we call specialization. A network function $f_{\theta }:{\mathbb{R}}^n\to {\mathbb{R}}^{|\tilde{T}|}$ can be turned into a ‘specialist’ for a new target classes ${\tilde{T}}^{\prime }=\{t_1,\dots ,t_m\}$, where each new target $t_i\in {\tilde{T}}^{\prime }$ is a proper subset of target classes $\tilde{T}$ of f such that $t_i\cap t_j=\varnothing$ if $t_i\ne t_j$.

Let G and t be the underlying graph of a neural network function and associated target vertex. By adding one more vertex u and adjoining it to the target vertex t of G, we can associate a function which is the composition of linear and activation functions along the arrow $\overrightarrow{a}$ whose source is t and the target is u, with u associated with ${\mathbb{R}}^m$. This results in a network function $g_{{\theta }^{\prime }}:{\mathbb{R}}^{|\tilde{T}|}\to {\mathbb{R}}^m$ with underlying graph $t\stackrel{\overrightarrow{a}}{⟶}u$. By composing f and g, we obtain ${\tilde{f}}_{({\theta }^{\prime },\theta )}=g_{{\theta }^{\prime }}\circ f_{\theta }$, whose target classes are $t_1,\dots ,t_m$, with the concatenated graph consisting of G and $t\stackrel{\overrightarrow{a}}{⟶}u$. Training the obtained network function ${\tilde{f}}_{({\theta }^{\prime },\theta )}$ is called a specialization.

4.5. Voting System

We assume the above setup and notations for a dataset X and a collection of trained models $\mathcal{M}$. We introduce a voting system that utilizes fuzzy domains and incorporates trained models with different targets. Pseudo-codes for the implementation are provided in Appendix A.

Let $T=\{c_1,\dots ,c_N\}$ be a given set of target classes, and suppose a measurable function $f:X\to T$ is given. In practice, this function is inferred from a statistical sample. We will compare f with the prediction obtained from $\mathcal{M}$.

Consider the subcollection of models $g_i$ in $\mathcal{M}$ which have flattened targets $T_i\subset T$. By abuse of notation, we still denote this subcollection by $\mathcal{M}$. We assume the following.

• If the flattened target $T_i$ of a model is minimal in the sense that $T_j\neg \subset T_i$ for any other $j\ne i$, then ${\tilde{T}}_i$ is fine, that is, all its elements are singleton.
• Every target classes $t_1,\dots ,t_k$ in a target set $\{t_1,\dots ,t_k\}$ has no intersection.
• $\mathcal{M}$ has a trained model whose flattened target equals $T=\{c_1,\dots ,c_N\}$.

Below, we first define a target graph, in which each node corresponds to a flattened target $T^{\prime }$. Next, we consider predictions from the collection of models with the same flattened target $T_i=T^{\prime }$ for some $T^{\prime }\subset T$. We then combine predictions from nodes along a path in a directed graph with no oriented cycle.

4.5.1. Construction of the Target Graph

We assign a partial order to the collection of trained models $\mathcal{M}$, where the weighted answers from each trained model accumulate according to this order. The partial order is encoded by a directed graph with no oriented cycle, which we call a target graph. Define $\mathcal{T}$ as the collection of flattenings, that is,

$$\mathcal{T}:=\left\{T_i:i\in \mathcal{I}\right\}.$$

Among the flattenings in $\mathcal{T}$, the partially ordered subset relation induces a directed graph that has a single source vertex, called the root node, which is associated with the given set of target classes $T=\{c_1,\dots ,c_N\}$.

Let $\mathfrak{T}$ denote the set of nodes in the target graph. For each node $s\in \mathfrak{T}$, let $T_s\in \mathcal{T}$ denote the associated flattening of the target, and define ${\mathcal{I}}_s$ as the index set

$${\mathcal{I}}_s:=\{i\in \mathcal{I}\mid T_i=T_s\},$$

which records the indices of the trained models corresponding to node s. By abuse of notation, $\mathfrak{T}$ also refers to the target graph itself, and $g_i\in {\mathcal{I}}_v$ indicates that $g_i$ is the trained model whose index i belongs to ${\mathcal{I}}_v$.

We define the refinement of targets, which allow us to systematically combine the predictions from multiple models at each node.

Definition 12.

Let T be a finite set and suppose we have a collection of subsets ${\tilde{T}}_i$ of the power set $\mathcal{P}\left(T\right)$ for $i\in \mathcal{I}$ such that for each i, its flattening equals T, that is, $\bigcup {\tilde{T}}_i=T$; moreover, $\varnothing \notin {\tilde{T}}_i$ and $t\cap t^{\prime }=\varnothing$ for all distinct $t,t^{\prime }\in {\tilde{T}}_i$. The common refinement is defined as

$$\left\{\bigcap _{i\in \mathcal{I}}{t}_i:{\left(t_i\right)}_{i\in \mathcal{I}}\in \prod _{i\in \mathcal{I}}{\tilde{T}}_i\right\}\setminus \left\{\varnothing \right\}.$$

At each node $v\in \mathfrak{T}$, we consider the collection ${\left\{{\tilde{T}}_i\right\}}_{i\in {\mathcal{I}}_v}$ of targets of all models at the node, and we take its common refinement ${\overline{T}}_v$. See Example 2.

4.5.2. Voting Rule for Multiple Models Sharing the Same Target

Let $I_E$ denote the characteristic function of a measurable set E, defined as

$$I_E\left(x\right)=\{\begin{array}{ll}1& \mathrm{if}x\in E\\ \hfill 0& \mathrm{otherwise},\end{array}$$

where $E\subset X$ or $E\subset {\mathbb{R}}^n$ for some positive integer n.

Let $(U_i,{\varphi }_i,g_i)$ be a triple consisting of trained model $g_i:{\mathbb{R}}^{n_i}\to {\tilde{T}}_i$, fuzzy subset $U_i\subset X$, and identification ${\varphi }_i:U_i\to {\mathcal{Z}}_i\subset {\mathbb{R}}^{n_i}\times {\tilde{T}}_i$. Let

$$\begin{array}{cc}\hfill {\widehat{x}}_i:=& {\pi }_1\circ {\varphi }_i\left(x\right)\in {\mathbb{R}}^{n_i},\hfill \\ \hfill {\widehat{y}}_i:=& {\pi }_2\circ {\varphi }_i\left(x\right)\in {\tilde{T}}_i\hfill \end{array}$$

denote the feature and output of realized data for each $x\in U_i$ via identification ${\varphi }_i$ with the projection maps ${\pi }_1$ and ${\pi }_2$ from ${\mathbb{R}}^{n_i}\times {\tilde{T}}_i$ onto their first and second components, respectively.

The accuracy function ${\Phi }_i$ of the trained model $g_i$ over ${\mathcal{Z}}_i$ at certainty threshold α is defined as

$${\Phi }_i\left(\alpha \right):={\displaystyle \frac{\left|\{({\widehat{x}}_i,{\widehat{y}}_i)\in {\mathcal{Z}}_{\alpha ,i}:P_i\left({\widehat{x}}_i\right)={\widehat{y}}_i\}\right|}{\left|{\mathcal{Z}}_{\alpha ,i}\right|}}.$$

Here, we denote the measure of a subset Z by $\left|Z\right|$.

Let $\mathcal{G}=\{g_1,\dots ,g_m\}$ be a family of trained models sharing the same target set $\tilde{T}=\{t_1,\dots ,t_k\}$ with accuracies ${\Phi }_1,\dots ,{\Phi }_m$, respectively. We define the weighted answer from $\mathcal{G}$, a group of trained models sharing the same targets for $x\in X$ at a certainty threshold α as

$${\mathbf{p}}_{\mathcal{G}}(\alpha ,x):=\left({\mathbf{p}}_{\mathcal{G}}{(\alpha ,x)}_1,\dots ,{\mathbf{p}}_{\mathcal{G}}{(\alpha ,x)}_k\right):={\left({\displaystyle \frac{{\sum }_i{I}_{X_{\alpha ,i}}\left(x\right){\Phi }_i\left(\alpha \right)g_{i,j}\left({\widehat{x}}_i\right)}{{\Phi }_{\mathcal{G}}\left(\alpha \right)}}\right)}_{j=1,\dots ,k},$$

where ${\Phi }_{\mathcal{G}}\left(\alpha \right):={\sum }_i{I}_{\{C_i\left(p_i\right)\ge \alpha \}}\left(p_i\right){\Phi }_i\left(\alpha \right)$ is the accuracy function of models in $\mathcal{G}$ with certainty threshold $\alpha$, and $X_{\alpha ,i}:={\varphi }^{-1}\left({\mathcal{Z}}_{\alpha ,i}\right)$ is the certain part of $g_i$ in X with certainty threshold $\alpha$ for each $i=1,\dots ,m$. If ${\Phi }_{\mathcal{G}}\left(\alpha \right)=0$, then we define ${\mathbf{p}}_{\mathcal{G}}(\alpha ,x)=\mathbf{0}\in {\mathbb{R}}^k$. See Example 2.

4.5.3. Voting Rule at a Node

Let v be a node in the target graph $\mathfrak{T}$ associated with flattened target $T_v$, refinements ${\overline{T}}_v$, and associated models ${\mathcal{I}}_v$. Consider the collection of all distinct target sets of models in ${\mathcal{I}}_v$, that is ${\left\{{\tilde{T}}_i\right\}}_{i\in {\mathcal{I}}_v}=\{{\tilde{T}}_{v,1},\dots ,{\tilde{T}}_{v,n_v}\}$ for some positive integer $n_v$. Let ${\mathcal{G}}_i$ denote the family of models sharing the same target ${\tilde{T}}_{v,i}=\{t_{i,1},\dots ,t_{i,k_i}\}$ for $i=1,\dots ,n_v$.

For each family of models ${\mathcal{G}}_i$, we have a combined answer vector $p_{{\mathcal{G}}_i}\in {\mathbb{R}}^{k_i}$ with the accuracy function ${\Phi }_{{\mathcal{G}}_i}$. Define ${\Psi }_{{\mathcal{G}}_i}$ as the weight function for the family of models $G_i$ as

$${\Psi }_{{\mathcal{G}}_i}:=\left\{\begin{array}{c}{\displaystyle \frac{{\Phi }_{{\mathcal{G}}_i}}{\sum {\Phi }_{{\mathcal{G}}_i}}}\mathrm{if}\sum {\Phi }_{{\mathcal{G}}_i}\ne 0,\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.$$

for each $i=1,\dots ,n_v$.

Since each ${\mathbf{p}}_{{\mathcal{G}}_i}{(\alpha ,x)}_j$, the j-th component of the weighted answer from ${\mathcal{G}}_i$ for x at certainty threshold $\alpha$ indicates how much it predicts x to be classified into target $t_{i,j}\in {\tilde{T}}_i$ at certainty threshold $\alpha$, we can multiply these ‘scores’ to compute the overall agreement on ${\cap }_{i=1}^{n_v}{t}_{i,j_i}$ among the families $G_i$. For a given tuple ${\left(t_{i,j_i}\right)}_i\in {\prod }_{i=1}^{n_v}{\tilde{T}}_{v,i}$ with multi-index $J=(j_1,\dots ,j_{n_v})$, define combined answer on $t_J$ (at node v) as

$$p_J(\alpha ,x)=\prod _{i=1}^{n_v}{\mathbf{p}}_{{\mathcal{G}}_i}{(\alpha ,x)}_{j_i},$$

where $t_J$ is the tuple ${\left(t_{i,j_i}\right)}_{i=1,\dots ,n_v}$. Let ${\mathcal{J}}_v$ be the set of all possible indices J.

Let ${\overline{T}}_v=\{{\overline{t}}_1,\dots ,{\overline{t}}_k\}$ be the collection of refinements at node v. Then there exist unique indices $J_1,\dots ,J_k$ such that ${\overline{t}}_s=\bigcap t_{J_s}:=t_{1,j_1}\cap \dots \cap t_{n_v,j_{n_v}}$, where $J_s=(j_1,\dots ,j_{n_v})$ for each $s=1,\dots ,k$. For each multi-index J in ${\mathcal{J}}_v$, we call J an invalid combination if $\cap t_J=\varnothing$ and a valid combination otherwise, that is, $\cap t_J\in {\overline{T}}_v$. For an invalid combination $J=(j_1,\dots ,j_{n_v})$, we define the contribution factor $\beta$ to distribute their combined answer $p_J$ to other refinements as follows:

$$\beta (t_{i,j_i},{\overline{t}}_s)=\left\{\begin{array}{cc}{\displaystyle \frac{p_{J_s}}{{\sum }_{{\overline{t}}_m\subset t_{i,j_i}}{p}_{J_m}}}\hfill & \mathrm{if}{\overline{t}}_s\subset t_{i,j_i}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

for each $i=1,\dots ,n_v$ and a valid refinement ${\overline{t}}_s\in {\overline{T}}_v$. Since $p_J=p_J·\left({\Psi }_{{\mathcal{G}}_1}+\dots +{\Psi }_{{\mathcal{G}}_{n_v}}\right)$, we can decompose $p_J$ into

$$\begin{array}{cc}\hfill p_J=& p_J{\Psi }_{{\mathcal{G}}_1}\beta (t_{1,j_1},{\overline{t}}_1)+\dots +p_J{\Psi }_{{\mathcal{G}}_{n_v}}\beta (t_{n_v,j_{n_v}},{\overline{t}}_1)\hfill \\ \hfill & +\dots +p_J{\Psi }_{{\mathcal{G}}_1}\beta (t_{1,j_1},{\overline{t}}_k)+\dots +p_J{\Psi }_{{\mathcal{G}}_{n_v}}\beta (t_{n_v,j_{n_v}},{\overline{t}}_k),\hfill \end{array}$$

as ${\sum }_{s=1}^k\beta (t_{i,j_i},{\overline{t}}_s)=1$ for any $t_{i,j_i}$. Then, we define $\mathcal{A}$ as the answer at node v, a function from X to ${\mathbb{R}}^k$ at certainty threshold $\alpha$, as

$${\mathcal{A}}_v={\left(p_{J_s}+\sum _{\begin{array}{c}J\in {\mathcal{J}}_{v,\mathrm{invalid}}\\ J=(j_1,\dots ,j_{n_v})\end{array}}\sum _{i=1}^{n_v}{p}_J{\Psi }_{{\mathcal{G}}_i}\beta (t_{i,j_i},{\overline{t}}_s)\right)}_{s=1,\dots ,k},$$

where ${\mathcal{J}}_{v,\mathrm{invalid}}$ is the collection of invalid combinations at node v. See Example 2.

Example 2

(An example of the voting procedure at a node). For a given node $v\in \mathfrak{T}$, let the flattened target $T_v=\{c_1,c_2,c_3,c_4,c_5\}$ and the indices of models ${\mathcal{I}}_v=\{(1,1),(1,2),2\}$ be associated with v. Suppose that the two models $g_{1,1}$ and $g_{1,2}$ share the target set ${\tilde{T}}_1$, and ${\tilde{T}}_2$ denotes the target set of $g_2$, where

$$\begin{array}{cc}\hfill {\tilde{T}}_1& =\{t_{1,1},t_{1,2}\}=\{\{c_1,c_2,c_3\},\{c_4,c_5\}\},\hfill \\ \hfill {\tilde{T}}_2& =\{t_{2,1},t_{2,2},t_{2,3}\}=\{\{c_1,c_2\},\{c_3,c_4\},\left\{c_5\right\}\}.\hfill \end{array}$$

Their refinement ${\overline{T}}_v$ is $\{\{c_1,c_2\},\left\{c_3\right\},\left\{c_4\right\},\left\{c_5\right\}\}$. Let ${\mathcal{G}}_1=\{g_{1,1},g_{1,2}\}$ and ${\mathcal{G}}_2=\left\{g_2\right\}$, the collections of models sharing the same targets.

Suppose that the accuracy functions ${\Phi }_{1,1},{\Phi }_{1,2},{\Phi }_2$ of models $g_{1,1},g_{1,2},g_2$ are given, respectively. For simplicity, we look at certainty threshold ${\alpha }_0=0$ and suppress our notation reserved for the certainty threshold α. Let an instance x be given, and trained models provide answers for x as follows:

$$\begin{array}{cccccc}\hfill g_{1,1}& =\left(a_{1,1},a_{1,2}\right),\hfill & \hfill g_{1,2}& =\left(a_{2,1},a_{2,2}\right),\hfill & \hfill g_2& =\left(b_1,b_2,b_3\right).\hfill \end{array}$$

Then, we have the weighted answers ${\mathbf{p}}_{\mathcal{G}}$ from each collection of models sharing the same targets $\mathcal{G}$ as follows:

$$\begin{array}{cc}\hfill {\mathbf{p}}_{{\mathcal{G}}_1}& =\left({\displaystyle \frac{{\Phi }_{1,1}{a}_{1,1}+{\Phi }_{1,2}{a}_{2,1}}{{\Phi }_{{\mathcal{G}}_1}}},{\displaystyle \frac{{\Phi }_{1,1}{a}_{1,2}+{\Phi }_{1,2}{a}_{2,2}}{{\Phi }_{{\mathcal{G}}_1}}}\right):=(a_1,a_2),\hfill \\ \hfill {\mathbf{p}}_{{\mathcal{G}}_2}& =\left(b_1,b_2,b_3\right),\hfill \end{array}$$

where ${\Phi }_{{\mathcal{G}}_1}={\Phi }_{1,1}+{\Phi }_{1,2}$ and ${\Phi }_{{\mathcal{G}}_1}={\Phi }_2$ are the accuracy functions of ${\mathcal{G}}_1$ and ${\mathcal{G}}_2$, respectively. Additionally, we have the weight functions ${\Psi }_{{\mathcal{G}}_1}={\displaystyle \frac{{\Phi }_{{\mathcal{G}}_1}}{{\Phi }_{{\mathcal{G}}_1}+{\Phi }_{{\mathcal{G}}_2}}}$ and ${\Psi }_{{\mathcal{G}}_2}={\displaystyle \frac{{\Phi }_{{\mathcal{G}}_2}}{{\Phi }_{{\mathcal{G}}_1}+{\Phi }_{{\mathcal{G}}_2}}}$.

Let ${\mathcal{J}}_v$, the collection of all combinations, be $\{J_1,J_2,J_3,J_4,J_5,J_6\}$, where

$$\begin{array}{cccccc}\hfill t_{J_1}& =\{t_{1,1},t_{2,1}\},\hfill & \hfill t_{J_2}& =\{t_{1,1},t_{2,2}\},\hfill & \hfill t_{J_3}& =\{t_{1,1},t_{2,3}\},\hfill \\ \hfill t_{J_4}& =\{t_{1,2},t_{2,1}\},\hfill & \hfill t_{J_5}& =\{t_{1,2},t_{2,2}\},\hfill & \hfill t_{J_6}& =\{t_{1,2},t_{2,3}\}.\hfill \end{array}$$

Note that $J_3$ and $J_4$ are invalid combinations at node v, and the refinements of valid combinations are $\cap t_{J_1}=\{c_1,c_2\},\cap t_{J_2}=\left\{c_3\right\},\cap t_{J_5}=\left\{c_4\right\}$, and $\cap t_{J_6}=\left\{c_5\right\}$. We compute the combined answer for each $t_{J_i}$ as

$$\begin{array}{cccccc}\hfill p_{J_1}=& a_1{b}_1,\hfill & \hfill p_{J_2}=& a_1{b}_2,\hfill & \hfill p_{J_3}=& a_1{b}_3,\hfill \\ \hfill p_{J_4}=& a_2{b}_1,\hfill & \hfill p_{J_5}=& a_2{b}_2,\hfill & \hfill p_{J_6}=& a_2{b}_3.\hfill \end{array}$$

Then, the nontrivial contribution factors β of $J_3$ defined in Equation (4) are

$$\begin{array}{cc}\hfill \beta (t_{1,1},\{c_1,c_2\})& ={\displaystyle \frac{p_{J_1}}{p_{J_1}+p_{J_2}}}={\displaystyle \frac{b_1}{b_1+b_2}},\hfill \\ \hfill \beta (t_{1,1},\left\{c_3\right\})& ={\displaystyle \frac{p_{J_2}}{p_{J_1}+p_{J_2}}}={\displaystyle \frac{b_2}{b_1+b_2}},\hfill \\ \hfill \beta (t_{2,3},\left\{c_5\right\})& =1,\hfill \end{array}$$

as $t_{1,1}=\{c_1,c_2,c_3\}$ and $t_{2,3}=\left\{c_5\right\}$, and β of $J_4$ are

$$\begin{array}{cc}\hfill \beta (t_{1,2},\left\{c_4\right\})& ={\displaystyle \frac{b_2}{b_2+b_3}},\hfill \\ \hfill \beta (t_{1,2},\left\{c_5\right\})& ={\displaystyle \frac{b_3}{b_2+b_3}},\hfill \\ \hfill \beta (t_{2,1},\{c_1,c_2\})& =1,\hfill \end{array}$$

as $t_{1,2}=\{c_4,c_5\}$ and $t_{2,1}=\{c_1,c_2\}$. Therefore, the answer ${\mathcal{A}}_v$ at node v for x is

$$\begin{array}{cc}\hfill {\mathcal{A}}_v\left(x\right)& =\left(a_1{b}_1+{\displaystyle \frac{a_1{b}_1{b}_3{\Psi }_{{\mathcal{G}}_1}}{b_1+b_2}}+a_2{b}_1{\Psi }_{{\mathcal{G}}_2},a_1{b}_2+{\displaystyle \frac{a_1{b}_2{b}_3{\Psi }_{{\mathcal{G}}_1}}{b_2+b_3}}\right.\hfill \\ \hfill & ,\left.a_2{b}_2+{\displaystyle \frac{a_2{b}_1{b}_2{\Psi }_{{\mathcal{G}}_1}}{b_2+b_3}},a_2{b}_3+a_1{b}_3{\Psi }_{{\mathcal{G}}_2}+{\displaystyle \frac{a_2{b}_1{b}_3{\Psi }_{{\mathcal{G}}_1}}{b_2+b_3}}\right).\hfill \end{array}$$

4.5.4. Accumulation of Votes Along Valid Paths

For a target class $c\in T$, a sequence of nodes $\gamma =(s_0,s_1,\dots ,s_m)$ in $\mathfrak{T}$ is called a valid path for c if $\gamma$ satisfies the following conditions:

• $s_0$ is the root of $\mathfrak{T}$.
• $c\in T_{s_i}$ for all $i=0,\dots ,m$.
• ${\overline{T}}_{s_m}$ consists of thin targets.

Let $\gamma =(s_0,s_1,\dots ,s_m)$ be a valid path for a class $c\in T$, where $s_0$ is the root of $\mathfrak{T}$. Since each trained model provides prediction independently, we define $\mathcal{M}(\gamma ,\alpha ,x)$, the weighted answer for x at certainty threshold α along a valid path γ, as the product

$$\mathcal{M}(\gamma ,\alpha ,x)={\left(\prod _{i=0}^m{A}_{s_i}{(\alpha ,x)}_{j_{i,t}}\right)}_{t\in {\overline{T}}_{s_m}}=\left({\mathcal{M}}_{t_1},\dots ,{\mathcal{M}}_{t_{|{\overline{T}}_{s_m}|}}\right),$$

(5)

which represents how much $\mathcal{M}$ predicts x to be classified in each target $t\in {\overline{T}}_{s_m}$ along the path $\gamma$. Here, $j_{i,t}$ is the index of $\overline{t}\in {\overline{T}}_{s_i}$ such that $\overline{t}$ is the the unique refinement in ${\overline{T}}_{s_i}$ containing c. Then, define $\mathbf{P}(\gamma ,\alpha ,x)$, the prediction for x at certainty threshold α along a valid path γ, as the $argmax\mathcal{M}(\gamma ,\alpha ,x)$.

Remark 9.

Under the specialization method explained in Section 4.4, we can construct the ‘gating layer’ as in the Mixture of Expert [11] using this voting strategy. Let g be a trained model in $\mathcal{M}$ and the targets of g be $\tilde{T}=\left\{t_1,\dots ,t_k\right\}$, where $t_i=\{c_{i,1},\dots ,c_{i,n_i}\}$ for $i=1,\dots ,k$ such that $T=\{c_{1,1},\dots ,c_{k,n_k}\}$. Then g serves as the ‘gating’ layer in $\mathcal{M}$ navigating an instance to other trained models that are trained on a dataset containing classes exclusively within a target $t_i$ of $\tilde{T}$. See [12] for the experimental results.

4.5.5. Vote Using Validation History

We introduce the ‘using validation history’ method in prediction to alleviate concerns regarding the optimal valid path or certainty threshold. In other words, $\alpha$ and $\gamma$ in Equation (5) are fixed through this method based on the validation dataset.

Let $X_{\mathrm{val}}$ be a measurable subset of X with $\mu \left(X_{\mathrm{val}}\right)<\infty$, where $\mu$ is the measure of X. Let $\mathrm{\Gamma }\left(c\right)$ denote the set of all valid paths in the target graph $\mathfrak{T}$ for a class c. Given the true label function $f:X\to T$, we define r as the expected accuracy along a path γ at certainty threshold α to class c as follows:

$$r_c(\gamma ,\alpha ):={\displaystyle \frac{\left|\left(\{\mathbf{P}(\gamma ,\alpha ,x)=c\}\cap f^{-1}\left(c\right)\right)\cup \{\mathbf{P}(\gamma ,\alpha ,x)\ne c\mathrm{and}f\left(x\right)\ne c\}\right|}{\left|X_{\mathrm{val}}\right|}},$$

where $\gamma \in \mathrm{\Gamma }\left(c\right)$. Since there is a finite number of valid paths for each target class and $\alpha \in [0,1]$, there exists a tuple of maximizers $({\gamma }^{*}\left(c\right),{\alpha }^{*}\left(c\right))$ for each class c such that the expected accuracy r attains its supremum at ${\alpha }^{*}\left(c\right)$. We define the answer using the validation history for $x\in X$ as

$$\mathcal{M}\left(x\right)=\left({\mathcal{M}}_{c_1}({\gamma }^{*}\left(c_1\right),{\alpha }^{*}\left(c_1\right),x),\dots ,{\mathcal{M}}_{c_N}({\gamma }^{*}\left(c_N\right),{\alpha }^{*}\left(c_N\right),x)\right).$$

Remark 10.

Let a system of trained models $\mathcal{M}$ and validation dataset $X_{\mathrm{val}}$ be given. The combining rule using the validation history for each trained model serves as the role of ρ in Definition 9, and $P:X\to [0,1]$ is defined by the vote of $\mathcal{M}$ using the validation history.

Remark 11.

To use the validation history, every model in the Logifold must first be evaluated on a validation dataset. Once this evaluation is completed, the time and space complexity of prediction become proportional to the number of models involved along the predetermined paths, which are bounded above by the total number of models in $\mathcal{M}$.

Constructing the target tree requires searching for partial subset relations under the assumption of a unique maximal element. In the worst case, this process has a complexity of $O\left(T^2\right)$, where T is the total number of target classes.

This bound arises because, in the worst case, the root may have $T-1$ children, such as $\{1,\dots ,T-1\}$, $\{1,\dots ,T-2,T\}$, …, $\{2,\dots ,T\}$. Each of these children can, in turn, have up to $T-2$ children, and so on. When a new model is inserted into the tree, its position must be determined by repeatedly checking subset relations at each level. At most, this search requires $1+(T-1)+(T-2)+\dots +2$ operations and memories, leading to a total complexity proportional to $O\left(T^2\right)$ in both time and space. Thus, constructing the full target tree or inserting a new model in the worst case involves quadratic growth in the number of target classes.

5. Experiments and Results

We summarize the results of two experiments from our previous paper [12] on well-known benchmark datasets as follows: CIFAR10 [13], MNIST [14], and Fashion MNIST [15].

The objective of first experiment was to demonstrate the logifold’s ability to mitigate the negative impact of poor-performing models within an ensemble. The setup involved combining six low-accuracy Simple CNN models (with an average accuracy of 56.45%) with one high-accuracy ResNet20 [40] model (achieving 85.96% accuracy) on the CIFAR10 test set (10,000 images). The logifold achieved a notable 84.86% accuracy, which outperformed simple averaging (62.55%) and majority voting (58.72%). This outcome underscores Logifold’s capability to leverage the contributions of the most well-trained component model, even when poor performance models are prevalent. Results are detailed in

Table 2. Accuracy results for varying certainty thresholds. Thresholds are selected as $\sigma (-\infty ),\sigma \left(0\right),\sigma \left(1\right),\dots ,\sigma \left(7\right)$, where $\sigma \left(x\right):={(1+e^{-x})}^{-1}$ denotes the sigmoid function. The second column indicates accuracy using the refined voting system without validation history; thus, the accuracy at the 0 threshold reflects standard weighted voting based on validation performance. The third column shows accuracy within the subset of data classified above each certainty threshold. Baseline comparisons (simple average and majority voting) and refined voting using validation history explained in Section 4.5.5 are summarized below the table.

Certainty Threshold	Accuracy by Refined Voting	Accuracy in Certain Part	The Number of Certain Data
0	0.6158	0.6158	10,000
0.5	0.6158	0.6158	10,000
0.7311	0.6187	0.6187	10,000
0.8808	0.7821	0.7821	10,000
0.9526	0.8185	0.8946	8653
0.9820	0.7856	0.9626	6906
0.9933	0.7363	0.9856	5361
0.9975	0.7068	0.9934	5361
0.9991	0.6740	0.9978	3246

Simple average: 0.6255. Majority voting: 0.5872. Voting system using validation history: 0.8486.

.

The second experiment highlights the flexibility of the Logifold structure when integrating models trained on diverse target datasets. We combined MNIST, Fashion MNIST, and CIFAR10 datasets by resizing MNIST and Fashion MNIST images from $28\times 28$ grayscale to $32\times 32$ RGB images, matching the CIFAR10 dataset dimensions. This concatenation created a unified dataset with 30 distinct target classes: 10 classes from each dataset. We trained a ResNet20 [40] model T on the combined dataset and ResNet20 models M, F, and C, each exclusively trained on MNIST, Fashion MNIST, and CIFAR10, respectively. Additionally, a filter model $T^{\prime }$ was derived through the specialization method explained in Section 4.4. Evaluated on a combined testing dataset of 30,000 samples (10,000 per dataset), the Logifold comprising $T^{\prime }$, M, F, and C achieved 89.31% accuracy using the refined voting with validation history method, outperforming that of a single model T at 75.16%. When model T was included alongside $T^{\prime }$, M, F, and C, the accuracy remained at 75.16% for simple averaging, weithed averaging or majority voting methods due to predictions of model T dominating these voting systems. However, refined voting system achieved 90.78%.

Table 3. Accuracy results for various Logifold compositions evaluated on the combined test set from MNIST, Fashion MNIST, and CIFAR10 datasets. The second column reports results obtained through non-refined voting methods (simple average, weighted average, or majority voting). The third column provides accuracy using refined voting with validation history.

Comprising Models	Accuracy by Non-Refined Voting	Accuracy by Refined Voting
T	75.16%	75.13%
$\left(M,F,C,T^{\prime }\right)$	9.93%	89.31%
$\left(T,M,F,C,T^{\prime }\right)$	75.16%	90.78%

summarizes the results.

6. Conclusions

The inherently discrete and often irregular nature of real-world data calls for a new mathematical framework that more faithfully models such complexity. Traditional manifold theory, while useful for interpolating data points, often falls short of addressing the essential discreteness and fuzziness needed in contemporary data science applications. Building on the local-to-global principle from classical geometry, we adapt it into a measure-theoretic context, deliberately moving beyond idealized assumptions like smoothness. We design frameworks that can handle the discrete, high-dimensional, and heterogeneous characteristics typical of modern datasets. Within this perspective, we propose the logifold structure (Definition 8)—a geometric construction that models the local behaviors of classifiers as “charts” and integrates them coherently into a global structure.

A key conceptual choice in this framework is to use the graphs of linear logical functions (Definition 1) as the fundamental local models. Instead of relying on a single global predictor, the logifold represents the data through a collection of these local, interpretable views—each similar in spirit to neural network models. Supported by the universality theorem (Theorem 5), this design ensures that the entire data space can be adequately covered by an atlas of specialized and diverse models. Each local model remains reliable within its domain, and together they form a coherent and mathematically grounded global ensemble. This research design, therefore, directly addresses the non-Euclidean, discrete, and high-dimensional challenges of real-world data by integrating well-understood local structures into a unified, flexible global framework.

We support our theoretical framework with an implementation of computer algorithms and experiments. In our previous work [12], we tested logifolds on the following well-known benchmark datasets: CIFAR10, MNIST, and Fashion MNIST. The experiments demonstrated that logifolds can effectively defend against poor outputs from underperforming models within an ensemble and flexibly combine models specialized for different target domains. Notably, the logifold ensemble substantially outperformed both simple averaging and majority voting baselines, and it also achieved higher accuracy than single models trained on the full task.

In future research, we shall focus on developing sophisticated, adaptive, and potentially learned methods for defining and enforcing model domains. This is essential for moving the logifold concept from a theoretical and experimental success toward broader practical adoption—particularly in settings involving adversarial attacks, out-of-distribution data, or highly dynamic environments where model uncertainty and reliability are critical.