Series of Semihypergroups of Time-Varying Artificial Neurons and Related Hyperstructures

Abstract

Detailed analysis of the function of multilayer perceptron (MLP) and its neurons together with the use of time-varying neurons allowed the authors to find an analogy with the use of structures of linear differential operators. This procedure allowed the construction of a group and a hypergroup of artificial neurons. In this article, focusing on semihyperstructures and using the above described procedure, the authors bring new insights into structures and hyperstructures of artificial neurons and their possible symmetric relations.

1. Introduction

As mentioned in the PhD thesis [1], neurons are the atoms of neural computation. Out of those simple computational units all neural networks are build up. The output computed by a neuron can be expressed using two functions $y=g\left(f\right(w,x\left)\right)$. The details of computation consist in several steps: In a first step the input to the neuron, $x:=\left\{x_i\right\}$, is associated with the weights of the neuron, $w:=\left\{w_i\right\}$, by involving the so-called propagation function f. This can be thought as computing the activation potential from the pre-synaptic activities. Then from that result the so-called activation function g computes the output of the neuron. The weights, which mimic synaptic strength, constitute the adjustable internal parameters of the neuron. The process of adapting the weights is called learning [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18].

From the biological point of view it is appropriate to use an integrative propagation function. Therefore, a convenient choice would be to use the weighted sum of the input $f(w,x)={\displaystyle \sum _i}{w}_i{x}_i$, that is the activation potential equal to the scalar product of input and weights. This is, in fact, the most popular propagation function since the dawn of neural computation. However, it is often used in a slightly different form:

$$f(w,x)={\displaystyle \sum _i}{w}_i{x}_i+\mathsf{\Theta }.$$

(1)

The special weight $\mathsf{\Theta }$ is called bias. Applying $\mathsf{\Theta }\left(x\right)=1$ for $x>0$ and $\mathsf{\Theta }\left(x\right)=0$ for $x<0$ as the above activation function yields the famous perceptron of Rosenblatt. In that case the function $\mathsf{\Theta }$ works as a threshold.

Let $F:\mathbb{R}\to \mathbb{R}$ be a general non-linear (or piece-wise linear) transfer function. Then the action of a neuron can be expressed by

$$y\left(k\right)=F\left({\displaystyle \sum _{i=1}^m}{w}_i\left(k\right)x_i\left(k\right)+b\right),$$

where $x_i\left(k\right)$ is input value in discrete time k where $i=0,\cdots ,m,w_i\left(k\right)$ is weight value in discrete time where $i=0,\cdots ,m,b$ is bias, $y_i\left(k\right)$ is output value in discrete time $k.$

Notice that in some very special cases the transfer function F can be also linear. Transfer function defines the properties of artificial neuron and this can be any mathematical function. Usually it is chosen on the basis of the problem that the artificial neuron (artificial neural network) needs to solve and in most cases it is taken (as mentioned above) from the following set of functions: step function, linear function and non-linear (sigmoid) function [1,2,5,7,9,12,16,19].

In what follows we will consider a certain generalization of classical artificial neurons mentioned above such that inputs $x_i$ and weight $w_i$ will be functions of an argument t belonging into a linearly ordered (tempus) set T with the least element $0$. As the index set we use the set $\mathbb{C}\left(J\right)$ of all continuous functions defined on an open interval $J\subset \mathbb{R}$. So, denote by W the set of all non-negative functions $w:T\to \mathbb{R}$ forming a subsemiring of the ring of all real functions of one real variable $x:\mathbb{R}\to \mathbb{R}$. Denote by $Ne({\overrightarrow{w}}_r)=Ne(w_{r1},\cdots ,w_{rn})$ for $r\in \mathbb{C}\left(J\right)$, $n\in \mathbb{N}$ the mapping

$$y_r\left(t\right)={\displaystyle \sum _{k=1}^n}{w}_{r,k}\left(t\right)x_{r,k}\left(t\right)+b_r$$

which will be called the artificial neuron with the bias $b_r\in \mathbb{R}.$ By $\mathbb{AN}\left(T\right)$ we denote the collection of all such artificial neurons.

Neurons are usually denoted by capital letters $X,Y$ or $X_i,Y_i$, nevertheless we use also notation $Ne(\overrightarrow{w})$, where $\overrightarrow{w}=(w_1,\cdots ,w_n)$ is the vector of weights [20,21,22].

We suppose - for the sake of simplicity - that transfer functions (activation functions) $\phi ,\sigma \left(orf\right)$ are the same for all neurons from the collection $\mathbb{AN}\left(T\right)$ and the role of this function plays the identity function $f\left(y\right)=y$.

Feedforward multilayer networks are architectures, where the neurons are assembled into layers, and the links between the layers go only into one direction, from the input layer to the output layer. There are no links between the neurons in the same layer. Also, there may be one or several hidden layers between the input and the output layer [5,9,16].

2. Preliminaries on Hyperstructures

From an algebraic point of view, it is useful to describe the terms and concepts used in the field of algebraic structures. A hypergroupoid is a pair $(H,·)$, where H is a (nonempty) set and

$$·:H\times H\to {\mathcal{P}}^{*}\left(H\right)(=\mathcal{P}\left(H\right)-\{\varnothing \})$$

is a binary hyperoperation on the set $H.$ If $a·(b·c)=(a·b)·c$ for all $a,b,c\in H$ (the associativity axiom), the the hypergroupoid $(H,·)$ is called a semihypergroup. A semihypergroup is said to be a hypergroup if the following axiom:

$$a·H=H=H·a$$

for all $a\in H$ (the reproduction axiom), is satisfied. Here, for sets $A,B\subseteq H,A\ne \varnothing \ne B$ we define as usually

$$A·B=\bigcup \{a·b;a\in A,b\in B\}.$$

Thus, hypergroups considered in this paper are hypergroups in the sense of F. Marty [23,24]. In some constructions it is useful to apply the following lemma (called also the Ends-lemma having many applications—cf. [25,26,27,28,29]). Recall, first that by a (quasi-)ordered semigroup we mean a triad $(S,·,\le ),$ where $(S,·)$ is a semigroup, $(S,\le )$ is a (quasi-)ordered set, i.e., a set S endowed with a reflexive and transitive binary relation “≤” and for all triads of elements $a,b,c\in S$ the implication $a\le b\Rightarrow a·c\le b·c,c·a\le c·b$ holds.

Lemma 1 (Ends-Lemma).

Let $(S,·,\le )$ be a (quasi-)ordered semigroup. Define a binary hyperoperation

$$\ast :S\times S\to {\mathcal{P}}^{*}\left(S\right)bya\ast b=\{x\in S;a·b\le x\}.$$

Then $(S,\ast )$ is a semihypergroup. Moreover, if the semigroup $(S,·)$ is commutative, then the semihypergroup $(S,\ast )$ is also commutative and if $(S,·,\le )$ is a (quasi-)ordered group then the semihypergroup $(S,\ast )$ is a hypergroup.

Notice, that if $(G,·),(H,·)$ are (semi-)hypergroups, then a mapping $h:G\to H$ is said to be the homomorphism of $(G,·)$ into $(H,·)$ if for any pair $a,b\in G$ we have

$$h(a·b)\subseteq h\left(a\right)·h\left(b\right).$$

If for any pair $a,b\in G$ the equality $h(a·b)=h\left(a\right)·h\left(b\right)$ holds, the homomorphism h is called the good (or strong) homomorphism—cf. [30,31]. By $EndG$ we denote the endomorphism monoid of a semigroup (group) $G.$

Concerning the basics of the hypergroup theory see also [23,25,26,27,28,32,33,34,35,36,37,38,39,40,41].

Linear differential operators described in the article and used e.g., in [29,42] are of the following form:

Definition 1.

Let $J\subseteq \mathbb{R}$ be an open interval, $\mathbb{C}\left(J\right)$ be the ring of all continuous functions $\phi :J\to \mathbb{R}.$ For $p_k\in \mathbb{C}\left(J\right),k=0,\cdots ,n-1,p_0\ne 0$ we define

$$L(p_{n-1},\cdots ,p_0)y\left(x\right)=y^{\left(n\right)}\left(x\right)+{\displaystyle \sum _{k=0}^{n-1}}{p}_k\left(x\right)y^{\left(k\right)}\left(x\right),y\in {\mathbb{C}}^n\left(J\right)$$

(the ring of all smooth functions up to order n, i.e., having derivatives up to order n defined on the interval $J\subseteq \mathbb{R}).$

Definition 2 

([41,49]). Let $(G,·)$ be a semigroup and $P\subset G,P\ne \varnothing .$ A hyperoperation ${\ast }^P:G\times G\to \mathcal{P}\left(G\right)$ defined by $[x,y]\to xPy,i.e.,x\ast y=xPy$ for any pair $[x,y]\in P\times P$ is said to be the P-hyperoperation in G. If

$$x{\ast }^P\left(y{\ast }^{P}z\right)=xPyPz=\left(x{\ast }^{P}y\right){\ast }^{P}z$$

holds for any triad $x,y,z\in G,$ the P-hyperoperation is associative. If also the axiom of reproduction is satisfied, the hypergrupoid $(G,{\ast }^P)$ is said to be a P-hypergroup.

Evidently, if $(G,·)$ is a group, then also $(G,{\ast }^P)$ is a P-hypergroup. If the set P is a singleton, then the P-operation ${\ast }^P$ is a usual single—valued operation.

Definition 3.

A subset $H\subset G$ is said to be a sub-P-hypergroup of $(G,{\ast }^P)$ if $P\subset H\subset G$ and $(H,{*}^P)$ is a hypergroup.

Now, similarly as in the case of the collection of linear differential operators [29], we will construct a group and hypergroup of artificial neurons, cf. [29,32,42,43,44].

Denote by ${\delta }_{ij}$ Kronecker delta, $i,j\in \mathbb{N},$ i.e., ${\delta }_{ii}={\delta }_{jj}=1$ and ${\delta }_{ij}=0$, whenever $i\ne j.$

Suppose $Ne({\overrightarrow{w}}_r),Ne({\overrightarrow{w}}_s)\in \mathbb{AN}\left(T\right),r,s\in \mathbb{C}\left(J\right),{\overrightarrow{w}}_r=(w_{r1},\cdots ,w_{r,n}),{\overrightarrow{w}}_s=(w_{s1},\cdots ,w_{s,n}),n\in \mathbb{N}.$ Let $m\in \mathbb{N},1\le m\le n$ be a such an integer that $w_{r,m}>0$. We define

$$Ne\left({\overrightarrow{w}}_r\right){·}_{m}Ne\left({\overrightarrow{w}}_s\right)=Ne\left({\overrightarrow{w}}_u\right),$$

where

$${\overrightarrow{w}}_u=(w_{u,1},\cdots ,w_{u,n})=(w_{u,1}\left(t\right),\cdots ,w_{u,n}\left(t\right)),$$

$${\overrightarrow{w}}_{u,k}\left(t\right)=w_{r,m}\left(t\right)w_{s,k}\left(t\right)+(1-{\delta }_{m,k})w_{r,k}\left(t\right),t\in T$$

and, of course, the neuron $Ne({\overrightarrow{w}}_u)$ is defined as the mapping $y_u\left(t\right)={\displaystyle \sum _{k=1}^n}{w}_k\left(t\right)x_k\left(t\right)+b_u,t\in T,$$b_u=b_r{b}_{s.}$ Further, for a pair $Ne({\overrightarrow{w}}_r),Ne({\overrightarrow{w}}_s)$ of neurons from $\mathbb{AN}\left(T\right)$ we put $Ne\left({\overrightarrow{w}}_r\right){\le }_{m}Ne\left({\overrightarrow{w}}_s\right),$$w_r=(w_{r,1}\left(t\right),\cdots ,w_{r,n}\left(t\right)),$$w_s=(w_{s,1}\left(t\right),\cdots ,w_{s,n}\left(t\right))$ if $w_{r,k}\left(t\right)\le w_{s,k}\left(t\right),k\in \mathbb{N},k\ne m$ and $w_{r,m}\left(t\right)=w_{s,m}\left(t\right),$$t\in T$ and with the same bias. Evidently $(\mathbb{AN}\left(T\right),{\le }_m)$ is an ordered set. A relationship (compatibility) of the binary operation "${·}_m$" and the ordering ${\le }_m$ on $\mathbb{AN}\left(T\right)$ is given by this assertion analogical to Lemma 2 in [29].

Lemma 2.

The triad $(\mathbb{AN}\left(T\right),{·}_m,{\le }_m)$ (algebraic structure with an ordering) is a non-commutative ordered group.

Sketch of the proof was published in [21].

Denoting

$${\mathbb{AN}}_1{\left(T\right)}_m=\{Ne\left(\overrightarrow{w}\right);\overrightarrow{w}=(w_1,\cdots ,w_n),w_k\in \mathbb{C}\left(T\right),k=1,\cdots ,n,w_m\left(t\right)\equiv 1\},1\le m\le n,$$

we get the following assertion:

Proposition 1 

(Prop. 1. [21], p. 239). Let $T=⟨0,t_0)\subset \mathbb{R},t_0\in \mathbb{R}\cup \{\infty \}$. Then for any positive integer $n\in \mathbb{N},n\ge 2$ and for any integer m such that $1\le m\le n$ the semigroup $({\mathbb{AN}}_1{\left(T\right)}_m,{·}_m)$ is an invariant subgroup of the group $(\mathbb{AN}{\left(T\right)}_m,{·}_m).$

Proposition 2 

(Prop. 2. [21], p. 240). Let $t_0\in \mathbb{R},t_0>0,T=⟨0,t_0)\subset \mathbb{R}$ and $m,,n\in \mathbb{N}$ are integers such that $1\le m\le n-1$. Define a mapping $F:{\mathbb{AN}}_n{\left(T\right)}_m\to {\mathbb{LA}}_n{\left(T\right)}_{m+1}$ by this rule: For an arbitrary neuron $Ne({\overrightarrow{w}}_r)\in {\mathbb{AN}}_n{\left(T\right)}_m,$ where ${\overrightarrow{w}}_r=(w_{r,1}\left(t\right),\cdots ,w_{r,n}\left(t\right))\in {\left[\mathbb{C}\left(T\right)\right]}^n$ we put $F(Ne({\overrightarrow{w}}_r))=L(w_{r,1},\cdots ,w_{r,n})\in {\mathbb{LA}}_n{\left(T\right)}_{m+1}$ with the action:

$$L(w_{r,1},\cdots ,w_{r,n})y\left(t\right)=\frac{d^{n}y\left(t\right)}{d{t}^n}+{\displaystyle \sum _{k=1}^n}{w}_{r,k}\left(t\right)\frac{d^{k-1}y\left(t\right)}{d{t}^{k-1}},y\in {\mathbb{C}}^n\left(T\right).$$

Then the mapping $F:{\mathbb{AN}}_n{\left(T\right)}_m\to {\mathbb{LA}}_n{\left(T\right)}_{m+1}$ is a homomorphism of the group $({\mathbb{AN}}_n{\left(T\right)}_m,{·}_{m)}$ into the group $({\mathbb{LA}}_n{\left(T\right)}_{m+1},{\circ }_{m+1}).$

Now, using the construction described in the Lemma 1, we obtain the final transposition hypergroup (called also non-commutative join space). Denote by $\mathbb{P}{(\mathbb{AN}{(T)}_m)}^{*}$ the power set of $\mathbb{AN}{(T)}_m$ consisting of all nonempty subsets of the last set and define a binary hyperoperation

$${*}_m:\mathbb{AN}{(T)}_m\times \mathbb{AN}{(T)}_m\to \mathbb{P}{(\mathbb{AN}{(T)}_m)}^{*}$$

by the rule

$$Ne({\overrightarrow{w}}_r){*}_{m}Ne({\overrightarrow{w}}_s)=\{Ne({\overrightarrow{w}}_u);Ne({\overrightarrow{w}}_r){·}_{m}Ne({\overrightarrow{w}}_s){\le }_{m}Ne({\overrightarrow{w}}_u)\}$$

for all pairs $Ne({\overrightarrow{w}}_r),Ne({\overrightarrow{w}}_s)\in \mathbb{AN}{(T)}_m$. More in detail if $\overrightarrow{w}(u)=(w_{u,1},\cdots ,w_{u,n}),\overrightarrow{w}(r)=(w_{r,1},\cdots ,w_{r,n}),\overrightarrow{w}(s)=(w_{s,1},\cdots ,w_{s,n})$, then $w_{r,m}(t)w_{s,m}(t)=w_{u,m}(t),w_{r,m}(t)w_{s,k}(t)+w_{r,k}(t)\le w_{u,k}(t),$ if $k\ne m,t\in T.$ Then we have that $(\mathbb{AN}{(T)}_m,{*}_m)$ is a non-commutative hypergroup. We say that this hypergroup is constructed by using the Ends Lemma (cf. e.g., [8,25,29]. These hypergroups can be called as EL-hypergroups. The above defined invariant (called also normal) subgroup $({\mathbb{AN}}_1{(T)}_m,{·}_m)$ of the group $(\mathbb{AN}{(T)}_m,{·}_m)$ is the carrier set of a subhypergroup of the hypergroup $(\mathbb{AN}{(T)}_m,{*}_m)$ and it has certain significant properties.

Using certain generalization of methods from [42] (p. 283), we obtain, after we investigate the constructed structures, the following result:

Theorem 1.

Let $T=⟨0,t_0)\subset \mathbb{R},t_0\in \mathbb{R}\cup \{\infty \}.$ Then for any positive integer $n\in \mathbb{N},n\ge 2$ and for any integer m such that $1\le m\le n$ the hypergroup $(\mathbb{AN}{(T)}_m,{*}_m)$,where

$$\mathbb{AN}{(T)}_m=\{Ne({\overrightarrow{w}}_r);{\overrightarrow{w}}_r=(w_{r,1}(t),\cdots ,w_{r,n}(t))\in {\left[\mathbb{C}(T)\right]}^n,w_{r,m}(t)>0,t\in T\},$$

is a transposition hypergroup (i.e., a non-commutative join space) such that $(\mathbb{AN}{(T)}_m,{*}_m)$ is its subhypergroup, which is

-

invertible (i.e., $Ne({\overrightarrow{w}}_r)/Ne({\overrightarrow{w}}_s)\cap {\mathbb{AN}}_1{(T)}_m\ne \varnothing$ implies $Ne({\overrightarrow{w}}_s)/Ne({\overrightarrow{w}}_r)\cap {\mathbb{AN}}_1{(T)}_m\ne \varnothing$ and $Ne({\overrightarrow{w}}_r)Ne({\overrightarrow{w}}_s)\cap {\mathbb{AN}}_1{(T)}_m\ne \varnothing$ implies $Ne({\overrightarrow{w}}_s)Ne({\overrightarrow{w}}_r)\cap {\mathbb{AN}}_1{(T)}_m\ne \varnothing$ for all pairs of neurons $Ne({\overrightarrow{w}}_r),Ne({\overrightarrow{w}}_s)\in {\mathbb{AN}}_1{(T)}_m,$

-

closed (i.e., $Ne({\overrightarrow{w}}_r)/Ne({\overrightarrow{w}}_s)\subset {\mathbb{AN}}_1{(T)}_m,Ne({\overrightarrow{w}}_r)\setminus Ne({\overrightarrow{w}}_s)\subset {\mathbb{AN}}_1{(T)}_m$ for all pairs $Ne({\overrightarrow{w}}_r),/,Ne({\overrightarrow{w}}_s)\in {\mathbb{AN}}_1{(T)}_m,$

-

reflexive (i.e., $Ne({\overrightarrow{w}}_r){\mathbb{AN}}_1{(T)}_m={\mathbb{AN}}_1{(T)}_m/Ne({\overrightarrow{w}}_r)$ for any neuron $Ne({\overrightarrow{w}}_r)\in \mathbb{AN}{(T)}_m$ and

-

normal (i.e.$Ne({\overrightarrow{w}}_r)\ast {\mathbb{AN}}_1{(T)}_m={\mathbb{AN}}_1{(T)}_m\ast Ne({\overrightarrow{w}}_r)$ for any neuron $Ne({\overrightarrow{w}}_r)\in \mathbb{AN}{(T)}_m.$

Remark 1.

A certain generalization of the formal (artificial) neuron can be obtained from expression of a linear differential operator of the n-th order. Recall the expression of formal neuron with inner potential $y_{-in}={\displaystyle \sum _{k=1}^n}{w}_k\left(t\right)x_k\left(t\right),$ where $\overrightarrow{x}\left(t\right)=(x_1\left(t\right),\cdots ,x_n\left(t\right))$ is the vector of inputs, $\overrightarrow{w}\left(t\right)=(w_1\left(t\right),\cdots ,w_n\left(t\right))$ is the vector of weights. Using the bias b of the considered neuron and the transfer function σ we can expressed the output as $y\left(t\right)=\sigma \left({\displaystyle \sum _{k=1}^n}{w}_k\left(t\right)x_k\left(t\right)+b\right).$

Now consider a tribal function $u:J\to \mathbb{R},$ where $J\subseteq \mathbb{R}$ is an open interval; inputs are derived from $u\in {\mathbb{C}}^n\left(J\right)$ as follows: Inputs $x_1\left(t\right)=u\left(t\right),x_2=\frac{du\left(t\right)}{dt},\cdots ,x_n\left(t\right)=\frac{d^{n-1}\left(t\right)}{d{t}^{n-1}},n\in \mathbb{N}.$ Further the bias $b=b_0\frac{d^{n}u\left(t\right)}{d{t}^n}.$ As weights we use the continuous functions $w_k:J\to \mathbb{R},k=1,\cdots ,n-1.$

Then formula

$$y\left(t\right)=\sigma \left({\displaystyle \sum _{k=1}^n}{w}_k\left(t\right)\frac{d^{k-1}u\left(t\right)}{d{t}^{k-1}}+b_0\frac{d^{n}u\left(t\right)}{d{t}^n}\right)$$

is a description of the action of the neuron $Dn$ which will be called a formal(artificial) differential neuron. This approach allows to use solution spaces of corresponding linear differential equations.

Proposition 3 

([41], p. 16). Let $(G_1,·),(G_2,·)$ be two groups $f\in Hom(G_1,G_2)$ and $P\subset G_1.$ Then the homomorphism f is a good homomorphism between P-hypergroups $(G_1,{\ast }^P)$ and $(G_2,{\ast }^P).$

Concerning the discussed theme see [26,27,28,30,32,36,39,45]. Now denote by $S\subseteq \mathbb{C}(T)$ an arbitrary non/empty subset and let

$$P=\{Ne({\overrightarrow{w}}_u(t));u\in S\}\subseteq \mathbb{AN}(T).$$

Then defining

$$Ne({\overrightarrow{w}}_p(t))\ast Ne({\overrightarrow{w}}_q(t))=Ne({\overrightarrow{w}}_p(t)){·}_{m}P{·}_{m}Ne({\overrightarrow{w}}_q(t))=$$

$$\{Ne({\overrightarrow{w}}_p(t)){·}_{m}Ne({\overrightarrow{w}}_u(t)){·}_{m}Ne({\overrightarrow{w}}_q(t));u\in S\}$$

for any pair of neurons $Ne({\overrightarrow{w}}_p(t)),Ne({\overrightarrow{w}}_q(t))\in \mathbb{AN}(T)$, we obtain a P-hypergroup of artificial time varying neurons. If S is a singleton, i.e., P is a one-element subset of $\mathbb{AN}(T)$, the obtained structure is a variant of $\mathbb{AN}(T)$. Notice, that any $f\in EndG$ for a group $(G,·)$ induces a good homomorphism of the P-hypergroups $(G,{\ast }^P),(G,{\ast }^{f(P)})$ and any automorphism creates an isomorphism beween the above P-hypergroups.

Let $(\mathbb{Z},+)$ be the additive group of all integers. Let $Ne({\overrightarrow{w}}_s(t))\in \mathbb{AN}(T)$ be arbitrary but fixed chosen artificial neuron with the output function $y_s(t)={\displaystyle \sum _{k=1}^n}{w}_{s,k}(t)x_{s,k}(t)+b_s$. Denote by ${\lambda }_s:\mathbb{AN}(T)\to \mathbb{AN}(T)$ the left translation within the group of time varying neurons determined by $Ne({\overrightarrow{w}}_s(t)$, i.e.,

$${\lambda }_s(Ne({\overrightarrow{w}}_p(t))=Ne({\overrightarrow{w}}_s(t){·}_{m}Ne({\overrightarrow{w}}_p(t))$$

for any neuron $Ne({\overrightarrow{w}}_p(t))\in \mathbb{AN}(T)$. Further, denote by ${\lambda }_s^r$ the r-th iteration of ${\lambda }_s$ for $r\in \mathbb{Z}$. Define the projection ${\pi }_s:\mathbb{AN}(T)\times \mathbb{Z}\to \mathbb{AN}(T)$ by

$${\pi }_s(Ne({\overrightarrow{w}}_p(t)),r)={\lambda }_s^r(Ne({\overrightarrow{w}}_p(t)).$$

It is easy to see that we get a usual (discrete) transformation group, i.e., the action of $(\mathbb{Z},+)$ (as the phase group) on the group $\mathbb{AN}(T)$. Thus the following two requirements are satisfied:

• ${\pi }_s(Ne({\overrightarrow{w}}_p(t)),0)=Ne({\overrightarrow{w}}_p(t))$ for any neuron $Ne({\overrightarrow{w}}_p(t))\in \mathbb{AN}(T)$,
• ${\pi }_s(Ne({\overrightarrow{w}}_p(t)),r+u)={\pi }_s({\pi }_s(Ne({\overrightarrow{w}}_p(t)),r),u)$ for any integers $r,u\in \mathbb{Z}$ and any artificial neuron $Ne({\overrightarrow{w}}_p(t))$. Notice that, in the dynamical system theory this structure is called a cascade.

On the phase set we will define a binary hyperoperation. For any pair of neurons $Ne({\overrightarrow{w}}_p(t)),$$Ne({\overrightarrow{w}}_q(t))$ define

$$Ne({\overrightarrow{w}}_p(t))\ast Ne({\overrightarrow{w}}_q(t))={\pi }_s(Ne({\overrightarrow{w}}_p(t)),\mathbb{Z})\cup {\pi }_s(Ne({\overrightarrow{w}}_q(t)),\mathbb{Z})=$$

$$\{{\lambda }_s^a(Ne({\overrightarrow{w}}_p(t));a\in \mathbb{Z}\}\cup \{{\lambda }_s^b(Ne({\overrightarrow{w}}_q(t));b\in \mathbb{Z}\}.$$

Then we have that $\ast :\mathbb{AN}(T)\times \mathbb{AN}(T)\to \mathcal{P}(\mathbb{AN}(T))$ is a commutative binary hyperoperation and since $Ne({\overrightarrow{w}}_p(t)),Ne({\overrightarrow{w}}_q(t))\in Ne({\overrightarrow{w}}_p(t))\ast Ne({\overrightarrow{w}}_q(t))$, we obtain that the hypergroupoid $(\mathbb{AN}(T),\ast )$ is a commutative, extensive hypergroup [20,27,29,30,31,34,35,38,43,46,47]. Using its properties we can characterize certain properties of the cascade $(\mathbb{AN}(T),\mathbb{Z},{\pi }_s).$ The hypergroup $(\mathbb{AN}(T),\ast )$ can be called phase hypergroup of the given cascade.

Recall now the concept of invariant subsets of the phase set of a cascade $(X,\mathbb{Z},{\pi }_s)$ and the concept of a critical point. A subset M of of a phase set X of the cascade $(X,\mathbb{Z},{\pi }_s)$ is called invariant whenever $\pi (x,r)\in M$, for all $x\in M$ and all $r\in \mathbb{Z}$. A critical point of a cascade is an invariant singleton. It is evident that a subset M of neurons, i.e., $M\subseteq \mathbb{AN}(T)$ is invariant in the cascade $(\mathbb{AN}(T),\mathbb{Z},{\pi }_s)$ whenever it is a carrier set of a subhypergroup of the hypergroup $(\mathbb{AN}(T),\ast )$, i.e., M is closed with respect to the hyperoperation ∗, which means $M\ast M={\displaystyle \bigcup _{a,b\in M}}a\ast b\subseteq M$. Moreover, union or intersection of an arbitrary non-empty system $\mathcal{M}\subseteq \mathbb{AN}(T)$ is also invariant.

3. Main Results

Now, we will construct series of groups and hypergroups of artificial neurons using certain analogy with series of groups of differential operators described in [29].

We denote by ${\mathbb{LA}}_n\left(J\right)$ (for an open interval $J\subseteq \mathbb{R}$) the set of all linear differential operators $L(p_{n-1},\cdots ,p_0),p_0\ne 0,p_k\in {\mathbb{C}}^n\left(J\right),$ i.e., the ring of all continuous functions defined on the interval J, acting as

$$L(p_{n-1},\cdots ,p_0)y\left(x\right)=y^n\left(x\right)+{\displaystyle \sum _{k=0}^{n-1}}{p}_k\left(x\right)y^k\left(x\right),y\in {\mathbb{C}}^n\left(J\right)$$

and endowed the binary operation

$$L(q_{n-1},\cdots ,q_0)\circ L(p_{n-1},\cdots ,p_0)=L(q_0{p}_{n-1}+q_{n-1},\cdots ,q_0{p}_1+q_1,q_0{p}_0).$$

Now denote by ${\overline{\mathbb{LA}}}_n\left(J\right)$ the set of all operators $\overline{L}(q_n,\cdots ,q_0),q_0\ne 0,q_k\in \mathbb{C}\left(J\right)$ acting as

$$\overline{L}(q_n,\cdots ,q_0)y\left(x\right)={\displaystyle \sum _{k=0}^n}{q}_k\left(x\right)y^{\left(k\right)}\left(x\right),q_0\ne 0,q_k\in \mathbb{C}\left(J\right)$$

with similarly defined binary operations such that ${\mathbb{LA}}_n\left(J\right),{\overline{\mathbb{LA}}}_n\left(J\right)$ are noncommutative groups. Define mappings $F_n:{\mathbb{LA}}_n\left(J\right)\to {\mathbb{LA}}_{n-1}\left(J\right)$ by

$$F_n\left(L(p_{n-1},\cdots ,p_0)\right)=L(p_{n-2},\cdots ,p_0)$$

and ${\varphi }_n:\mathbb{LA}\left(J\right)\to {\overline{\mathbb{LA}}}_{n-1}\left(J\right)$ by

$${\varphi }_n\left(L(p_{n-1},\cdots ,p_0)\right)=\overline{L}(p_{n-2},\cdots ,p_0).$$

It can be easily verified that both $F_n$ and ${\varphi }_n,$ for an arbitrary $n\in \mathbb{N}$, are group homomorphisms.

Evidently, ${\mathbb{LA}}_n\left(J\right)\subset {\overline{\mathbb{LA}}}_n\left(J\right),{\overline{\mathbb{LA}}}_{n-1}\left(J\right)\subset {\overline{\mathbb{LA}}}_n\left(J\right)$ for all $n\in \mathbb{N}$. Thus we obtain complete sequences of ordinary linear differential operators with linking homomorphisms $F_n,{\varphi }_n:$

Now consider the groups of time-varying neurons $(\mathbb{AN}{\left(T\right)}_m,{·}_m)$ from Proposition 3 and above defined homomorphism of the group $({\mathbb{AN}}_n{\left(T\right)}_m,{·}_m)$ into the group $({\mathbb{LA}}_n{\left(T\right)}_{m+1},{\circ }_{m+1})$. Then we can change the diagram in the following way:

Using the Ends lemma and results the theory of linear operators we can describe also mapping morphisms in sequences groups of linear differential operators:

$$\begin{array}{c}{\mathbb{LA}}_n\left(J\right)\underset{F_{n+1}}{\leftarrow }{\mathbb{LA}}_{n+1}\left(J\right)\underset{F_{n+2}}{\leftarrow }{\mathbb{LA}}_{n+2}\left(J\right)\underset{F_{n+3}}{\leftarrow }{\mathbb{LA}}_{n+3}\left(J\right)\underset{F_{n+4}}{\leftarrow }\cdots \\ \mathrm{as}\mathrm{so}\mathrm{analogy}\mathrm{in}\mathrm{sequences}\mathrm{groups}\mathrm{of}\mathrm{time-varying}\mathrm{neurons}:\\ {\mathbb{AN}}_n{\left(T\right)}_m\underset{F_{n+1}^{*}}{\leftarrow }{\mathbb{AN}}_{n+1}{\left(T\right)}_m\underset{F_{n+2}^{*}}{\leftarrow }{\mathbb{AN}}_{n+2}{\left(T\right)}_m\underset{F_{n+3}^{*}}{\leftarrow }{\mathbb{AN}}_{n+3}{\left(T\right)}_m\underset{F_{n+4}^{*}}{\leftarrow }\cdots \end{array}$$

(2)

Theorem 2.

Let $T=⟨0,t_0)\subset \mathbb{R},t_0\in \mathbb{R}\cup \{\infty \}$, $n\in \mathbb{N}$ such that $n\ge 2,m\in \mathbb{N}$ such that $m\le n$. Let $({\mathbb{HAN}}_n{\left(T\right)}_m,{*}_m)$ be the hypergroup obtained from the group $({\mathbb{AN}}_n{\left(T\right)}_m,{\circ }_m)$ by Proposition 2. Suppose that $F_n:({\mathbb{AN}}_n{\left(T\right)}_m,{\circ }_m)\to ({\mathbb{AN}}_{n-1}{\left(T\right)}_m,{\circ }_m)$ are the above defined surjective group homomorphisms. Then $F_n:({\mathbb{HAN}}_n{\left(T\right)}_m,{*}_m)\to ({\mathbb{HAN}}_{n-1}{\left(J\right)}_m,{*}_m)$ are surjective homomorphisms of hypergroups.

Remark 2.

The second sequence of (2) can thus be bijectively mapped onto sequence of hypergroups

$${\mathbb{HAN}}_n{\left(T\right)}_m\underset{F_{n+1}}{\leftarrow }{\mathbb{HAN}}_{n+1}{\left(T\right)}_m\underset{F_{n+2}}{\leftarrow }{\mathbb{HAN}}_{n+2}{\left(T\right)}_m\underset{F_{n+3}}{\leftarrow }{\mathbb{HAN}}_{n+3}{\left(T\right)}_m\underset{F_{n+1}}{\leftarrow }\cdots$$

wit the linking surjective homomorphisms $F_n$. Therefore, the bijective mapping of the above mentioned sequences is functorial.

Now, shift to the concept of an automaton. This was developed as a mathematical interpretation of real-life systems that work on a discrete time-scale. Using the binary operation of concatenation of chains of input symbols we obtain automata with input alphabets endowed with the structure of a semigroup or a group. Considering mainly the structure given by transition function and neglecting output functions with output sets we reach a very useful generalization of the concept of automaton called quasi—automaton [29,31,48,49]. Let us introduce the concept of automata as an action of time varying neurons. Moreover, let system $(A,S,\delta )$, consists of nonempty time-varying neuron set of states $A\subseteq \mathbb{AN}{\left(T\right)}_m$, arbitrary semigroup of their inputs S and let mapping $\delta :A\times S\to A$ fulfill the following condition:

$$\delta (\delta (a,r),s)=\delta (a,r\dot{s})$$

for arbitrary $a\in A$ and $r,s\in S$ can be understood as a analogy of concept of quasi-automaton, as a generalization of the Mealy-type automaton. The above condition is some times called Mixed Associativity Condition (MAC).

Definition 4.

Let A be a nonempty set, $(H,·)$ a semihypergroup and $\delta :A\times H\to A$ a mapping satisfying the condition:

$$\delta (\delta (s,a),b)\in \delta (s,a\dot{b})\left(3\right)$$

for any triad $(s,a,b)\in A\times H\times H,$ where $\delta (s,a\dot{b})=\{\delta (s,x);x\in a·b\}$. The triad $(A,H,\delta )$ is called a quasi-multiautomaton with the state set A and the input semihypergroup $(H,·)$. The mapping $\delta :A\times H\to A$ is called transition function (or next-state function) of the quasi-multiautomaton $(A,H,\delta )$. Condition $\left(3\right)$ is called Generalized Mixed Associativity Condition (or GMAC).

The just defined structures are also called as actions of semihypergroups $(H,·)$ on sets A (called state sets).

Neuron $Ne(\overrightarrow{w})$ acts as described above:

$$y\left(t\right)={\displaystyle \sum _{i=1}^n}{w}_i\left(t\right)x_i\left(t\right)+b,$$

where i goes from 0 to $n,$$w_i\left(t\right)$ is the weight value in continuous time, b is a bias and $y\left(t\right)$ is the output value in continuous time $t$. Here the transition function F is the identity function.

Now suppose that the input functions $x_i$ are differentiable up to arbitrary order $n$.

We consider linear differential operators

$$L(m,w_n,\cdots ,w_0):{\mathbb{C}}^n\left(T\right)\times \cdots \times {\mathbb{C}}^n\left(T\right)\to {\mathbb{C}}^n\left(T\right),i.e.{\mathbb{C}}^n\left(T\right)\times \cdots \times {\mathbb{C}}^n\left(T\right)={\left[{\mathbb{C}}^n\left(T\right)\right]}^{n+1},$$

defined

$$L(m,w_n,\cdots ,w_0)x\left(t\right)=$$

$$=mb+{\displaystyle \sum _{k=1}^n}{w}_k\left(t\right)\frac{d^k{x}_k\left(t\right)}{d{t}^k},x\left(t\right)=(x_0\left(t\right),x_1\left(t\right),\cdots ,x_n\left(t\right))\in {\mathbb{C}}^n\left(T\right)\times \cdots \times {\mathbb{C}}^n\left(T\right)={\left[{\mathbb{C}}^n\left(T\right)\right]}^{n+1}.$$

Then we denote by $\mathbb{L}N{e}_n\left(T\right)$ the additive Abelian group of linear differential operators $L(m,w_n,\cdots ,w_0)$, where for $L(m,w_n,\cdots ,w_0),L(k,w_n^{*},\cdots ,w_0^{*})\in \mathbb{L}N{e}_n\left(T\right)$ with the bias b we define

$$L(m,w_n,\cdots ,w_0)+L(s,w_n^{*}\cdots ,w_0^{*})=L(m+s,w_n+w_n^{*},\cdots ,w_0+w_0^{*}),$$

where

$$L(m+s,w_n+w_n^{*},\cdots ,w_0+w_0^{*})x\left(t\right)=$$

$$=(m+s)b+{\displaystyle \sum _{k=0}^n}(w_k\left(t\right)+w_k^{*}\left(t\right))\frac{d^k{x}_k\left(t\right)}{d{t}^k},t\in Tandx\left(t\right)=(x_0\left(t\right),x_1\left(t\right),\cdots ,x_n\left(t\right))\in {\left[{\mathbb{C}}^n\left(T\right)\right]}^{n+1}.$$

Suppose that $w_k\left(t\right)\in {\mathbb{C}}^n\left(T\right)$ and define

$${\delta }_n:{\mathbb{C}}^n\left(T\right)\times \mathbb{L}N{e}_n\left(T\right)\to {\mathbb{C}}^n\left(T\right)$$

by

$${\delta }_n(x\left(t\right),L(m,w_n,\cdots ,w_0))=mb+x\left(t\right)+m+{\displaystyle \sum _{k=0}^n}{w}_k\left(t\right)\frac{d^{k}x\left(t\right)}{d{t}^k},x\left(t\right)\in {\mathbb{C}}^n\left(T\right),where$$

$w_n,\cdots ,w_0$ are weights corresponding with inputs and b is the bias of a neuron corresponding to the operator $L(m,w_n,\cdots ,w_0)\in \mathbb{L}N{e}_n\left(T\right)$.

Theorem 3.

Let $\mathbb{L}N{e}_n\left(T\right),{\mathbb{C}}^n\left(T\right)$ be the above defined structures and ${\delta }_n:{\mathbb{C}}^n\left(T\right)\times \mathbb{L}N{e}_n\left(T\right)\to {\mathbb{C}}^n\left(T\right)$ be the above defined mapping. Then the triad $({\mathbb{C}}^n\left(T\right),\mathbb{L}N{e}_n\left(T\right),{\delta }_n)$ is an action of the group $\mathbb{L}N{e}_n\left(T\right)$ on the group ${\mathbb{C}}^n\left(T\right)$, i.e., a quasi-automaton with the state space ${\mathbb{C}}^n\left(T\right)$ and with the alphabet $\mathbb{L}N{e}_n\left(T\right)$ with the group structure of artificial neurons.

Proof. 

We are going to verify the mixed associativity condition (MAC). Suppose $x\in {\mathbb{C}}^n\left(T\right)$ and $L(m,w_n,\cdots ,w_0),L(k,u_n,\cdots ,u_0)\in \mathbb{L}N{e}_n\left(T\right)$. Then we have

$${\delta }_n({\delta }_n(x\left(t\right),L(m,w_n,\cdots ,w_0),L(k,u_n,\cdots ,u_0))=$$

$$={\delta }_n(mb+x\left(t\right)+m+{\displaystyle \sum _{k=0}^n}{w}_k\left(t\right)\frac{d^{k}x\left(t\right)}{d{t}^k},L(k,u_n,\cdots ,u_0))=$$

$$=kb+mb+x\left(t\right)+m+k+{\displaystyle \sum _{k=0}^n}{w}_k\left(t\right)\frac{d^{k}x\left(t\right)}{d{t}^k}+{\displaystyle \sum _{k=0}^n}{u}_k\left(t\right)\frac{d^{k}x\left(t\right)}{d{t}^k}=$$

$$=(m+k)b+x\left(t\right)+m+k+{\displaystyle \sum _{k=0}^n}(w_k\left(t\right)+u_k\left(t\right))\frac{d^{k}x\left(t\right)}{d{t}^k}=$$

$${\delta }_n(x\left(t\right),L(m+k,w_n\left(t\right)+u_n\left(t\right),\cdots ,w_0)\left(t\right)+u_0)\left(t\right))=$$

$${\delta }_n(x\left(t\right),L(m,w_n,\cdots ,w_0)+L(k,u_n,\cdots ,u_0)),$$

thus MAC is satisfied. □

Consider an interval $T\subseteq \mathbb{R}$ and the ring $\mathbb{C}\left(T\right)$ of all continuous functions defined on the interval. Let $\{{\phi }_k;k\in \mathbb{N}\}$ be a sequence of ring-endomorphism of $\mathbb{C}\left(T\right)$. Denote ${\mathbb{A}}_{n+k}\mathbb{N}{\left(T\right)}_m$ the EL-hypergroup of artificial neurons constructed above, with vectors of weights of the dimension $n+k\in \mathbb{N}(=\{1,2,3\cdots \left\}\right)$. Let ${\left[\mathbb{C}\left(T\right)\right]}^{n+k}{=\mathbb{C}\left(T\right)\times \mathbb{C}\left(T\right)\times \cdots \times \mathbb{C}\left(T\right)(n+k-times)i.e.,\mathbb{C}\left(T\right)]}^{n+k}$ is the $n+k$-dimensional cartesian cube. Denote by ${\overline{\phi }}_k:{\left[\mathbb{C}\left(T\right)\right]}^{n+k}\to {\left[\mathbb{C}\left(T\right)\right]}^{n+k-1}$ the extension of ${\phi }_k$ such that ${\overline{\phi }}_k(\overrightarrow{w})={\overline{\phi }}_k\left((w_1,\cdots ,w_{n+k-1},w_{n+k})\right)=(w_1,\cdots ,w_{n+k-1})$. Let us denote the mapping $F_k:{\mathbb{A}}_{n+k}\mathbb{N}{\left(T\right)}_m\to {\mathbb{A}}_{n+k-1}\mathbb{N}{\left(T\right)}_m$ defined by $F_k(Ne(\overrightarrow{w})=Ne({\overrightarrow{w}}_1)$ with ${\overrightarrow{w}}_1=(w_1,\cdots ,w_{n+k})$. Consider underlying sets of hypergroups ${\mathbb{A}}_{n+k}\mathbb{N}{\left(T\right)}_m$ endowed with the above defined ordering relation:

$$for\overrightarrow{w}=(w_1,\cdots ,w_{n+k}),\overrightarrow{u}=(u_1,\cdots ,u_{n+k})\in {\left[\mathbb{C}\left(T\right)\right]}^{n+k}$$

we hanve $\overrightarrow{w}\le \overrightarrow{u}$ if $w_r\le u_r,r=1,2,\cdots ,n+kand{w}_m\le u_m$. Now, for $Ne(\overrightarrow{w}),Ne(\overrightarrow{u})\in {\mathbb{A}}_{n+k}\mathbb{N}{\left(T\right)}_m$ such that $\overrightarrow{w}=(w_1,\cdots ,w_{n+k}),\overrightarrow{u}=(u_1,\cdots ,u_{n+k}),Ne(\overrightarrow{w})\le Ne(\overrightarrow{u}),$ which means $\overrightarrow{w}\le \overrightarrow{u}$ ($w_m=u_m$ and biases of corresponding neurons are the same) we have ${\overline{\phi }}_k(\overrightarrow{w})=(w_1,\cdots ,w_{n+k-1})\le (u_1,\cdots ,u_{n+k-1})={\overline{\phi }}_k(\overrightarrow{u}),$ which implies $F_k(\overrightarrow{w})\le F_k(\overrightarrow{u})$.

Consequently the mapping $F_k:({\mathbb{A}}_{n+k}\mathbb{N}{\left(T\right)}_m,\le )\to ({\mathbb{A}}_{n+k-1}\mathbb{N}{\left(T\right)}_m,\le )$ is order-preserving, i.e., this is an order-homomorphism of hypergroups. The final result of our considerations is the following sequence of hypergroups of artificial neurons and linking homomorphisms:

$${\mathbb{A}}_n\mathbb{N}{\left(T\right)}_m\stackrel{F_1}{\leftarrow }{\mathbb{A}}_{n+1}\mathbb{N}{\left(T\right)}_m\stackrel{F_2}{\leftarrow }\cdots \stackrel{F_K}{\leftarrow }{\mathbb{A}}_{n+k}\mathbb{N}{\left(T\right)}_m\stackrel{F_{K+1}}{\leftarrow }{\mathbb{A}}_{n+k+1}\mathbb{N}{\left(T\right)}_m\cdots$$

4. Conclusions

Artificial neural networks and structured systems of artificial neurons have been discussed by a great number of researchers. They are an important part of artificial intelligence with many useful applications in various branches of science and technical constructions. Our considerations are based on algebraic and analytic approach using certain formal similarity with classical structures and new hyperstructures of differential operators. We discussed a certain generalizations of classical artificial time-varying neurons and studied them using recently derived methods. The presented investigations allow further development.