This article is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
A multi-layered perceptron type neural network is presented and analyzed in this paper. All neuronal parameters such as input, output, action potential and connection weight are encoded by quaternions, which are a class of hypercomplex number system. Local analytic condition is imposed on the activation function in updating neurons’ states in order to construct learning algorithm for this network. An error back-propagation algorithm is introduced for modifying the connection weights of the network.
Processing multi-dimensional data is an important problem for artificial neural networks. A single neuron can take only one real value as its input, thus a network should be configured so that several neurons are used for accepting multi-dimensional data. This type of configuration is sometimes unnatural in applications of artificial neural networks to engineering problem, such as processing of acoustic signals or coordinates in the plane. Thus, complex number systems have been utilized to represent two-dimensional data elements as a single entity. Application of complex numbers to neural networks have been extensively investigated, as summarized in the references [1,2,3].
Though complex values can treat two-dimensional data elements as a single entity, what we should treat data with more than two-dimension in artificial neural networks? Although this problem can of course be solved by applying several real-valued or complex-valued neurons, it would be useful to introduce a number system with higher dimensions, the so-called hypercomplex number systems.
Quaternion is a four-dimensional hypercomplex number system introduced by Hamilton [4,5]. This number system has been extensively employed in several fields, such as modern mathematics, physics, control of satellites, computer graphics, etc. [6,7,8]. One of the benefits provided by quaternions is that affine transformations of geometric figures in three-dimensional spaces, especially spatial rotations, can be represented compactly and efficiently. Applying quaternions to the field of neural networks has been recently explored in an effort to naturally represent high-dimensional information, such as color and three-dimensional body coordinates, by a quaternionic neuron, rather than complex-valued or real-valued neurons.
Thus, there has been a growing number of studies concerning the use of quaternions in neural networks. Multilayer perceptron (MLP) models have been developed in [9,10,11,12,13,14]. The use of quaternion in MLP models has been applied to several engineering problems such as control problems [10], color image compression [12], color night vision [15,16], and predictions for the output of chaos circuits and winds in three-dimensional space [13,14]. Other types of network models has also been explored, such as the computational ability of a single quaternionic neuron [17] and the existence condition of an energy function in continuous-time and continuous-state recurrent networks [18]. There are several types of quaternionic Hopfield-type networks with discrete-time driven, such as bipolar state [19,20], continuous state [21,22], multistate by phase representation [23,24]. Learning schemes for these networks have also been proposed in [25].
One of the difficulties in constructing neural networks in the quaternionic domain is about the introduction of suitable functions for the activation function in updating the neurons’ states. A typical type of activation function is the so-called “split” type function, in which a real-valued function is appliedto update each component of a quaternionic value [10]. Real-valued sigmoidal function and hyperbolic function, which are analytic (differentiable) functions, are often used for this purpose. However, as an activation function, the split-type quaternionic function is not appropriate due to lack of analyticity. Thus, it is necessary to define other types of differential functions in order to construct learning schemes such as the error back-propagation algorithm. The Cauthy-Riemann-Fueter (CRF) equation defines the analytic condition for the quaternionic functions, which corresponds to the Cauthy-Riemann equation for the complex-valued functions. The functions satisfying the CRF equation turn out to be only linear functionsor constants; therefore it is impossible to introduce non-linearity into the updates of the neurons’ states.
Recently, another class of analyticity for the quaternionic functions has been developed [26,27,28]. Called “local analyticity”, this analytic condition is derived at a quaternionic point with its local coordinate, rather than in a quaternionic space with a global coordinate. The derivation of local analytic condition shows that a quaternion in the local coordinate system is isomorphic to the complex number system and thus it can be treated as a complex value. A neural network with an activation function with local analyticity has been first proposed and analyzed in [13,14] in terms of MLP-type network. It is shown that the performance for this network is superior to the network with a split-type activation function through several applications. Another application of analytic activation function to quaternionic neural networks has been investigated in terms of Hopfield-type network [22]. The local analytic condition is constructed based on [28], and the stability conditions are derived in the case of quaternionic tanh function being used for an activation function is deduced in this network, in which the complex-valued tanh function used in [29] can be used as a quaternionic function.
This paper presents an MLP-type quaternionic neural network with locally analytic activation function. All variables in this network, such as input, output, action potential and connection weights, are encoded by quaternions. A learning scheme, a quaternionic equivalent of error back-propagation algorithm, is presented and theoretically explored. The derivation of the learning scheme in this papera dopts the Wirtinger calculus [30], which has been invented in the field of complex analysis, where a quaternionic value and its conjugate are treated independent of each other. This calculus enables the derivations to be more straightforward of its description, than by using the conventional description, i.e., Cartesian representation.
2. Quaternionic Algebra2.1. Definition of Quaternion
Quaternions forma class of hypercomplex numbers consistingofareal number and three imaginary numbers-i, j, and k. Formally, a quaternion number is defined as a vector x in a four-dimensional vector space,
where x(e), x(i), x(j), and x(k) are real numbers. The division ring of quaternions, H, constitutes the four-dimensional vector space over the real numbers with bases 1, i, j, and k. Equation (1) can also be written using 4-tuple or 2-tuple notation as
where = { x(i), x(j), x(k)}. In this representation, x(e) is the scalar part of x, and forms the vector part. The quaternion conjugate is defined as
Quaternion bases satisfy the following identities,
known as the Hamilton rule. From these rules, it follows immediately that multiplication of quaternions is not commutative.
Next, we define the operations between quaternions p = (p(e), ) = (p(e), p(i), p(j), p(k)) and q = (q(e), ) = (q(e), q(i), q(j), q(k)). The addition and subtraction of quaternions are defined in a similar manner as for complex-valued numbers or vectors, i.e.,
The product of p and q is determined by Equation (5) as
where · and × denote the dot and cross products, respectively, between three-dimensional vectors and . The conjugate of the product is given as
The quaternion norm of x, denoted by |x|, is defined as
2.2. Quaternionic Analyticity
It is important to introduce an analytic function (or differentiable function) to serve as the activation function in the neural network. This section describes the required analyticity of the function in the quaternionic domain, in order to construct activation functions for quaternionic neural networks.
The condition for differentiability of the quaternionic function f is given by
The analytic condition for the quaternionic function, called the Cauchy-Riemann-Fueter (CRF) equation, yields:
This is an extension of the Cauchy-Riemann (CR) equations defined for the complex domain. However, only linear functions and constants satisfy the CRF equation [14,26,27].
An alternative approach to assure analyticity in the quaternionic domain has been explored in [26,27,28]. This approach is called local analyticity and is distinguished from the standard analyticity, i.e., global analyticity. In the following, we introduce local derivatives with Wirtinger representation and analytic conditions for quaternionic functions, with reference to [28].
A quaternion x can be alternatively represented as:
From the definition in Equation (15), we deduce that u2 x = −1. If ux holds a commutative property against a difference of x, then the system with ux can be regarded as locally isomorphic to the complex number system.
A quaternionic difference of x, denoted by dx =(dx(e), dx(i), dx(j), dx(k)), can be decomposed by using:
where,
Then, the following relations hold:
when we set dx⊥ = 0, i.e., dx + uxdxux = 0,which results in uxdx = dxux. This leads to ux × d =0, because ux is a quaternion without a real part. Thus, ux and d are parallel to each other. Then, d = uxδ can be obtained, where δ is a real-valued constant. From Equation (14), it follows that
Then,
Considering = uxr and d = uxδ, we obtain
Hence, δ = dr is derived and d = uxdr is obtained. dx is represented as .
Figure 1 shows a schematic coordinate system for defining a local complex plane. The component k is omitted (x(k) = 0) in this figure due to difficulties in representing a four-dimensional vector space. In this example, for a given quaternion x, its unit vector ux is defined in the i-j plane. Then, a complex plane is defined by spanning the components x(e)(real axis) and x(r) in the quaternionic space, and the analytic condition is constrained in this plane.
A schematic illustration of local complex plane in a quaternionic space, where the component k is omitted for simplicity.
The local derivative operators are introduced, corresponding to the form of dx‖, as follows:
where
with the properties
Note that the variables x and x∗ turn out to be independent of each other. These derivative operators are quaternionic equivalents to the well-known Wirtinger derivative in the complex domain [30].
F (x + dx) can be expanded using the above-mentioned representations as
where,
When dx⊥ = 0, the local derivative of F (x) is written as
and the local analytic condition for the function F (x) is given by
in the corresponding local complex plane. This result corresponds to the one presented in [27], where dx⊥ =0 always holds.
Moreover, if F is a function with the two arguments, x and x∗, it becomes:
with x and x∗ being independent of each other. As a result, we can treat quaternionic functions in the same manner as complex-valued functions under the condition of local analyticity.
3. Quaternionic Multilayer Perceptron3.1. Network Model
The structure of the network assumed in this paper is shown in Figure 2. This network is a so-called multilayer perceptron network with one hidden layer, and the parameters in the network are encoded by quaternionic values.
The numbers of neurons in the input, hidden and output layers are set to M, N and K, respectively. A set of quaternionic signals denoted by z is input to the neurons in the input layer of the network. The outputs of the neurons in the input layer are the same as input z’s. In the hidden layer, each neuron takes the weighted sum of the output signals from the input layer. The (connection) weight from the n-th neuron in the input layer to the m-th neuron in the hidden layer is denoted by vnm. The output of the neuron in the hidden layer, denoted by xn, is determined by
where g is a quaternionic activation function introducing non-linearity between the action potential and output in the neuron. This function satisfies the following condition:
The structure of the multilayer perceptron in this paper.
Processing the neurons’ outputs in the output layer can be defined in the same manner as in the hidden layer. The output of the neuron in the output layer, yk, is defined as
where the function h is a quaternionic activation function from the action potential to the output, and wkn is the connection weight between the n-th neuron in the hidden layer and the k-th neuron in the output layer. The function h also satisfies
The connection weights should be modified by the so-called learning algorithms, in order to obtain the desired output signals with respect to the input signals. One of the learning algorithms for MLP-type networks is the error back-propagation (EBP) algorithm. The following section describes our derivation of this algorithm for the presented network.
3.2. Learning Algorithm
An EBP algorithm works so that the output error, calculated by the neurons’ outputs at the output layer and the desired output signals, is minimized. In the case of networks with three layers as shown in Figure 2, the connection weights between the hidden and output layers are first modified, and then the weights between the input and hidden layers are modified. In general (networks with n layers), EBP algorithms first modify the connection weights between the n-th layer (the output layer) and (n − 1)-th layer, and then between the (n − 1)-th layer and the (n − 2)-th layer, etc. This section only describes the three-layers case.
First, let dk be a quaternionic desired signal for the k-th output neuron when z’s are input to the network. The connection weights affect the output signals with respect to a set of input signals, thus the error E is regarded as a function with arguments wkn’s and wkn ∗’s. The output error E at the time t is then defined as
The output error should be real-valued so that it can be minimized.
Suppose that the connection weights are updated at the time (t + 1) by
where Δwkn is a quantity in updating. Then, the output error at the time (t + 1) can be written as
Note that the local analytic condition in quaternionic domain should be satisfied in calculating the derivatives. Thus, if we set Δwkn as
where µ is a quaternionic constant, the temporal difference of the output error, ∆E, becomes
If the real part of µ is positive, ∆E ≤ 0 holds. This indicates that the output error would decrease upon updating weights according to Equations (24) and (26). For calculating the updated quantity in Equation (26), the component ∂E/∂wkn∗‖ are expanded by using chain rule of derivative and ∂y/∂w∗‖= 0 from the local analytic condition is applied:
where h′ is the (local) derivative of the activation function h, and δk is defined as .
Similarly, the updates for the connection weights v’s can be deduced. The output error function E isa function with arguments vnm’s and v∗nm’s, thus the output error at the time (t + 1) can be represented by
Hence, vnm is updated with the quantity ∆vnm,
This leads to ∆E ≤ 0 under the condition of the real part of µ being positive. The component ∂E/∂*vnm‖ can be expanded by chain rules and local analytic conditions, ∂x/∂v*‖ = 0, ∂y/∂x*‖ = 0, and ∂x/∂v*‖ = 0 are applied:
Using the derivatives , and , we finally obtain
Once a set of network output {yk} is obtained for a set of net work input, the output error with respect to a target set {dk} can be calculated by Equation (23). Then, the connection weights between hidden and output layers are modified by Equations (24-28). The connection weights between input and hidden layers are finally modified by Equations (30-32).
3.3. Universal Approximation Capability
As an example of activation functions for neurons (g and h), the quaternionic tanh function [22] can be used. Other types of activation functions are also available, because complex-valued functions can be used for the presented network and the properties of several functions have been explored for the activation functions in [29]. It is important to consider the capability of the proposed quaternionic network with these activation functions, i.e., whether the proposed network can approximate given functions.
This concern is known as the universal approximation theorem [31,32]. In the real-valued MLPs with single hidden layer,the universality has been proven with a so-called sigmoidal function being used as an activation function of neurons. Other than sigmoidal functions, the bounded and differentiable functions are also available for activation functions.
This theorem is also discussed in the case of complex-valued networks [29]. The condition for boundedness in the real-valued MLPs is not required in some complex-valued MLPs. There are three types of activation functions discussed in [29], which are categorized by the properties of complex-valued functions, and for each of them, it is shown that universal approximation can be achieved.
The first type of complex-valued functions concerns the functions without anysingular points. These functions can be used as activation functions and the networks with this type of activation functions are shown as good approximators. Although some of the functions are not bounded, they can be used by introducing bounding operation for their regions. The second type concerns the functions having the bounded singular points, e.g., the discontinuous functions. These singularities can be removed and thus they can also be used for activation functions and can achieve their universality. The last type is for the functions with the so-called essential singularities, i.e., their singularities cannot be removed. These functions can also be used as activation functions, with the consideration of restricting the regions for them so that their regions never cover their singularities.
In the proposed quaternionic MLPs, for example, a quaternionic tanh function can be used as an activation function. This function is unbounded and may contain several kinds of singularities as in the case of complex-valued functions described above. Thus this quaternionic MLPs would face the same problem, i.e., the existence of singularities, but this can also be handled similarly as in the case of complex-valued MLPs for the removal or avoidance of such singularities. It could be possible to show the universality with handling singularities for the proposed MLPs according to the ways adopted in the complex-valued MLPs [29],but it remains as our future work.
4. Conclusions and Discussion
This paper has proposed a multilayer type neural network and an error back-propagation algorithm for its learning scheme in the quaternionic domain. The neurons in this network adopt locally analytic activation functions. The quaternionic functions with local analytic conditions are isomorphic to the complex functions, thus several activation functions, such as complex-valued tanh function, can be used extendedly in the quaternionic domain. The Wirtinger calculus, where a quaternion and its conjugate are treated as independent of each other, makes the derivation of the learning scheme clear and compact.
Analytic conditions for quaternionic functions are derived by defining a complex plane at a quaternionic point, which is a kind of reduction from quaternionic domain to complex domain. There exists another type of reduction in quaternionic domains, such as the commutative quaternion, which is a four-dimensional hypercomplex number system with commutativity in its multiplication [33,34,35,36]. A principal property is that a commutative quaternion can be decomposed and represented by two complex numbers with two linearly independent bases (called the decomposed form by idempotent bases). Commutative quaternions have been applied to neural networks only in terms of Hopfield-type network [24], but defining multilayer perceptron type networks can alsobe straightforward. Thus, it will be interesting to explore the relationship between commutative quaternion-based networks and networks with the local analyticity.
Showing the universality of the proposed network is also an important issue. Quaternionic functions, such as tanh function, may contain several kinds of singularities where the values of functions or their differentials are not defined in particular regions. In complex-valued networks with fully complex-valued function [29], the universality of the networks can be shown by dealing with these singularities, so that such singularities are removed or avoided by restricting the regions. It is expected to show the universality of the proposed network, in a similar way to the case of complex-valued networks.
Also, it is necessary to investigate the performances of the proposed network, though in this paper the experimental exploration could not be accomplished. The proposed network is similar to the networks proposed in [13,14], due to the introduction of local analytic function in quaternionic domain, thus the performances for both types of networks may have similar tendencies. Performance comparisons can also be conducted between the proposed network and the ones in [29], because both networks adopt the same representation in their constructions, i.e., Wirtinger calculus. A wide variety of activation functions have been investigated including the split-type function, phase-preserving function [37], and circular-type function [38]. Similar experiments should be conducted for the quaternionic networks.
Application of the presented network to engineering problems is also challenging. The processing of three or four dimensional vector data, such as color/multi-spectral image processing, predictions for three-dimensional protein structures, and controls of motion in three-dimensional space, will be the candidates from now on.
Acknowledgments
This study was financially supported by Japan Society for the Promotion of Science (Grant-in-Aids for Young Scientists (B) 24700227 and Scientific Research (C) 23500286).
ReferencesHiroseA.Complex-Valued Neural Networks: Theories and ApplicationWorld Scientific PublishingSingapore2003HiroseA.Complex-Valued Neural NetworksSpringer-VerlagBerlin, Germany2006NittaT.Complex-Valued Neural Networks: Utilizing High-Dimensional ParametersInformation Science ReferenceNew York, NY, USA2009HamiltonW.R.Lectures on QuaternionsHodges and SmithDublin, Ireland1853HankinsT.L.Sir William Rowan HamiltonJohns Hopkins University PressBaltimore, MD, USA1980MukundanR.Quaternions: From classical mechanics to computer graphics, and beyondProceedings of the 7th Asian Technology Conference in MathematicsMelaka, Malaysia17-21 December 200297105KuipersJ.B.Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace and Virtual RealityPrinceton University PressPrinceton, NJ, USA1998HoggarS.G.Mathematics for Computer GraphicsCambridge University PressCambridge, MA, USA1992NittaT.An extension of the back-propagation algorithm to quaternionsProceedings of International Conference on Neural Information Processing (ICONIP’96)Hong Kong, China24-27 September 19961247250ArenaP.FortunaL.MuscatoG.XibiliaM.Multilayer perceptronsto approximate quaternion valued functionsNeural Netw.19971033534210.1016/S0893-6080(96)00048-2BuchholzS.SommerG.Quaternionic spinor MLPProceeding of 8th European Symposium on Artificial Neural Networks (ESANN 2000)Bruges, Belgium26-28 April 2000377382MatsuiN.IsokawaT.KusamichiH.PeperF.NishimuraH.Quaternion neural network with geometrical operatorsJ. Intell. Fuzzy Syst.200415149164MandicD.P.JahanchahiC.TookC.C.A quaternion gradient operator and its applicationsIEEE Signal Proc. Lett.201118475010.1109/LSP.2010.2091126UjangB.C.TookC.C.MandicD.P.Quaternion-valued nonlinear adaptive filteringIEEE Trans. Neural Netw.2011221193120610.1109/TNN.2011.2157358KusamichiH.IsokawaT.MatsuiN.OgawaY.MaedaK.Anewschemeforcolornight vision by quaternion neural networkProceedings of the 2nd International Conferenceon Autonomous Robots and Agents (ICARA2004)Palmerston North, New Zealand13-15 December 2004101106IsokawaT.MatsuiN.NishimuraH.Quaternionic neural networks: Fundamental properties and applicationsComplex-Valued Neural Networks: Utilizing High-Dimensional ParametersNittaT.Information Science ReferenceNew York, NY, USA2009411439Chapter XVINittaT.A solution to the 4-bit parity problem with a single quaternary neuronNeural Inf. Process. Lett. Rev.200453339YoshidaM.KuroeY.MoriT.Models of hopfield-type quaternion neural networks and their energy functionsInt. J. Neural Syst.20051512913510.1142/S012906570500013XIsokawaT.NishimuraH.KamiuraN.MatsuiN.Fundamental properties of quaternionic hopfield neural networkProceedings of 2006 International Joint Conference on Neural NetworksVancouver BC, USA30 October 2006610615IsokawaT.NishimuraH.KamiuraN.MatsuiN.Associative memoryin quaternionic hopfield neural networkInt. J. Neural Syst.20081813514510.1142/S0129065708001440IsokawaT.NishimuraH.KamiuraN.MatsuiN.Dynamics of discrete-time quaternionic hopfield neural networksProceedings of 17th International Conference on Artificial Neural NetworksPorto, Portugal9-13 September 2007848857IsokawaT.NishimuraH.MatsuiN.On the fundamental properties of fully quaternionic hopfield networkProceedings of IEEE World Congress on Computational Intelligence (WCCI2012)Brisbane, Australia10-15 June 201212461249IsokawaT.NishimuraH.SaitohA.KamiuraN.MatsuiN.On the scheme of quaternionic multistate hopfield neural networkProceedings of Joint 4th International Conference on Soft Computing and Intelligent Systems and 9th International Symposium on Advanced Intelligent Systems (SCIS&ISIS 2008)Nagoya, Japan17-21 September 2008809813IsokawaT.NishimuraH.MatsuiN.Commutative quaternion and multistate hopfield neural networksProceedings of IEEE World Congress on Computational Intelligence (WCCI2010)Barcelona, Spain18-23 July 201012811286IsokawaT.NishimuraH.MatsuiN.An iterative learning schemefor multistate complex-valued and quaternionic hopfield neural networksProceedings of International Joint Conference on Neural Networks (IJCNN2009)Atlanta, GA, USA14-19 June 200913651371LeoS.D.RotelliP.P.Local hypercomplex analyticity1997(accessed on 20 November 2012)Available online:http://arxiv.org/abs/funct-an/9703002LeoS.D.RotelliP.P.Quaternonic analyticityAppl. Math. Lett.2003161077108110.1016/S0893-9659(03)90097-8SchwartzC.Calculus with a quaternionic variableJ. Math. Phys.200950013523:1013523:11KimT.AdalıT.Approximationby fully complex multilayer perceptronsNeural Comput.2003151641166610.1162/089976603321891846WirtingerW.Zur formalen theorie der funktionen von mehr komplexen veränderlichenMath. Ann.19279735737510.1007/BF01447872CybenkoG.Approximations by superpositions of sigmoidal functionsMath. Control Signals Syst.1989230331410.1007/BF02551274HornikK.Approximation capabilities of multilayer feedforward networksNeural Netw.19914215257SegreC.The real representations of complex elements and extension to bicomplex systemsMath. Ann.189240322335CatoniF.CannataR.ZampettiP.An Introduction to commutative quaternionsAdv. Appl. CliffordAlgebras20061612810.1007/s00006-006-0002-yDavenportC.M.A commutative hypercomplex algebra with associated function theoryClifford Algebra With Numeric and Symbolic ComputationAblamowiczR.BirkhauserBoston, MA, USA1996213227PeiS.C.ChangJ.H.DingJ.J.Commutative reduced biquaternions and their Fourier Transformfor signal and image processing applicationsIEEE Trans. Signal Proc.2004522012203110.1109/TSP.2004.828901HiroseA.Continuous complex-valued back-propagation learningElectron. Lett.1992281854185510.1049/el:19921186GeorgiouG.M.KoutsougerasC.Complex domain backpropagationIEEE Trans. Circuits Syst. II19923933033410.1109/82.142037