Method of increasing the information capacity of associative memory of oscillator neural networks using high-order synchronization effect

Computational modelling of two- and three-oscillator schemes with thermally coupled $VO_2$-switches is used to demonstrate a novel method of pattern storage and recognition in an impulse oscillator neural network (ONN) based on the high-order synchronization effect. The method ensures high information capacity of associative memory, i.e. a large number of synchronous states $N_s$. Each state in the system is characterized by the synchronization order determined as the ratio of harmonics number at the common synchronization frequency. The modelling demonstrates attainment of $N_s$ of several orders both for a three-oscillator scheme $N_s$~650 and for a two-oscillator scheme $N_s$~260. A number of regularities are obtained, in particular, an optimal strength of oscillator coupling is revealed when $N_s$ has a maximum. A general tendency toward information capacity decrease is shown when the coupling strength and switch inner noise amplitude increase. An algorithm of pattern storage and test vector recognition is suggested. It is also shown that the coordinate number in each vector should be one less than the switch number to reduce recognition ambiguity. The demonstrated method of associative memory realization is a general one and it may be applied in ONNs with various mechanisms and oscillator coupling topology.


Introduction
Usage of artificial neural networks [1] for information processing allows mastering the problems that arise when traditional computation schemes are applied in such areas as pattern and speech recognition [2].Therefore the important research trends include studying the modes of oscillator neural networks (ONN) operation and training, implementation of associative memory modes based, for example, on weakly coupled phase oscillators (Kuramoto model) [3] or impulse oscillators [4].The effect of synchronization plays a crucial role in ONN operation and is often used as a marker of ONN action, for example, in pattern recognition event.
In ONN the system demonstrates frequency and phase synchronization [5] - [8] and also synchronization of high order [9] - [11] at certain control parameters such as parameters of an oscillator scheme or coupling strengths between the oscillators.The proposed here method of pattern storage and recognition is based on high-order synchronization.In a lot of studies [8], [12], [13] patterns to be stored are expressed through a set of vectors.Vector coordinates contain information about the pattern and unambiguously associate it with one of possible variants.
There are some methods of associative memory realization based on oscillator elements synchronization in ONN, one of them is presented in paper [5].To store E vector a phase-shift keying method of encoding is used in a fully connected massive of N oscillators and the vector itself is specified by a certain combination of phase shift E=(1, 2…N) using a weight matrix according to Habbian rule.Such method of (phase) encoding enables storage of more than one vector at the same weight matrix, i.e. there are several stable phase states of a system.Maximal number of stored vectors will be determined by the number of oscillators N. The recognition procedure has two stages, at the first one a test vector T is specified by weight matrix setting, at the second stage the weights are sharply changed to the initial values (corresponding to the stored vectors) and the system arrives at one of stable combination of phase shift E.However, this phase method has the following drawbacks: N 2 couplings with tunable weights and a two-stage procedure of pattern recognition.
A second known method of associative memory realization is a frequency-shift keying method of encoding based on synchronized frequency shift [8].According to this method vector E is stored through oscillator frequencies shift against the central frequency of oscillator array F 0 synchronization (on the first harmonic) per the values corresponding to the vector coordinates E=( 1 ,  2 … N ).Recognition of test vector T occurs at the reverse shift of frequencies and in case when the vectors coincide TE the synchronization indicating the fact of pattern recognition takes place.This method allows usage of an oscillator star configuration and only N couplings, however, the disadvantage of this method is that just one vector is stored.
The present paper suggests a breakthrough method of associative memory realization when the array of coupled oscillators allows storage and recognition of variety of patterns and their number (information capacity of storage) depends on the oscillator number N, range of control parameters, network topology, strength of coupling between oscillators and noise level in the system.This result is achieved because we use high order synchronization effect in our ONN that possesses plenty of oscillator synchronous states defined as harmonic number ratio at common frequency of synchronization.

Method 2.1 General principle
Oscillator neural network is a system of N coupled oscillators, they may be connected via electric (by resistors and capacitors) [14], [15], thermal [10], [11] and optic [16] couplings depending on the physical mechanism of oscillator interaction.In the general case there is a matrix of coupling strengths i,j (weights), where i, j are the numbers of interacting oscillators and i,j denotes the value of the i-th oscillator effect on the j-th one.Oscillator network may form various topologies: fully connectedall-toall; and not fully connectedbus, star, and ring.Fig. 1a-c show examples of two and three oscillators connections using topologies "star", "all-to-all" and an example of N oscillators connection using a mixed topology (Fig. 1d) 2 3 Fig. 1.Examples of two (a) and three (b) oscillators connection into a neural network using topologies "star" (b), "all-to-all" (c), and N-oscillators using mixed topology (d), where i,j indicates the value of the i-th oscillator effect on the j-th one.
It is known [9] - [11] that oscillators in a network may undergo the effect of synchronization and besides synchronization on the first harmonic synchronous modes of high order may be observed if signal spectra possess several harmonics.In the general case high-order synchronization is determined by the ratio k1:k2:k3:..kN, where kN is harmonic order of N-th oscillator at the common frequency of the network synchronization Fs.As an example Fig 2a shows spectra of three electric oscillators that have synchronization of the order k1:k2:k3=3:6:4.The following rule should be noted: if all paired oscillators have different synchronization frequencies there is always a common synchronization frequency Fs for the whole system (all pairs) and the network synchronous state will also be determined by the ratio k1:k2:k3:..kN at frequency Fs (see section 2.2)  b) schematic illustration of the synchronization areas for a threeoscillator scheme, c) example of vector and image association, where Ip is current of a power source, F 0 is first harmonic, k is the harmonic number at the synchronization frequency Fs.
Transition from one synchronous state into another is possible when the oscillator network control parameters are varied.For example, in electric oscillators the main parameters may be oscillator feed currents Ip, their variation causes changes of the basic oscillation frequency F 0 .Nevertheless, in some cases, transition between states may be achieved by variation of coupling forces or noise intensity.
The range of control parameters variation where synchronization does not change its state is called a synchronization area.There is a whole family of synchronization areas that are called Arnold's tongues (for the case of two oscillators).A schematic example of synchronization areas for a three-oscillator scheme is shown in Fig. 2b.Here the control parameters are oscillator feed currents.
The number of possible variants of synchronous states (synchronization areas) where the system may exist when the basic control parameters are varied we shall call information capacity of associative memory Ns.The value of Ns depends on the oscillator number N, control parameters range, network topology, strength of coupling between oscillators and noise level in the system.We will cover the issue in detail later, nevertheless, we have shown in [10] that for a two-oscillator network Ns has a maximum at certain values of coupling strengths between oscillators and decreases when the system noise amplitude increases.When the coupling strength grows considerably the value Ns decreases because of the nearby synchronization areas integration.
As we have mentioned in the introduction section, the patterns to be stored are usually expressed through a set of vectors.Vector coordinates contain information about the pattern and unambiguously associate it with one of possible variants (see example in Fig. 2c).
We suggest the following method of associative memory realization in a neural network based on high-order synchronization effect.
First, it is necessary to set the neural network at such initial point of control parameters (S) that is outside the system's synchronization areas (see Fig. 2b).While in a non-synchronous state the system may potentially reach one of various synchronization states (Ns) along the corresponding vectors E. Thus the system stores Ns vectors at any point outside the synchronous areas, we shall denote them as E1, E2...ENS.The dimension of the stored vectors in this case is equal to the oscillator number N and their coordinates determine the shift of oscillator control parameters, for instance, currents E1=(Ip1, Ip2…IpN), against the initial point S. Thus to store the certain vectors it is necessary to set the system initial point S in such non-synchronous state that the distances from this point to the existing synchronous states corresponded the stored vectors E.
Second, when the initial point is set, to recognize the test vector T it is necessary to apply the shifts equal to the vector's coordinates to the control parameters.If one of the conditions is met (T=E1 or T=E2… or T=ENS), i.e. coordinates values of T are equal to one of the stored patterns, a transition to the synchronous state will occur and, actually, the act of the corresponding pattern recognition will take place.Existence of synchronization areas ensures pattern recognition even at insignificant deviation of its coordinates from the stored pattern.
This method is a more complicated version of the method described in [8] where the analogy to the frequency-shift keying method of coding is used, just here the vector is specified through the control parameters E1=(Ip1, Ip2…IpN) instead of frequencies E=(1, 2…N), but the meaning is the same.The main difference of our method is in application of high-order synchronization effect that enables us to increase information capacity of associative memory of a neural network by many times.
Besides, as described in the results section, it is more practical to use vector dimensions E'=(I1, I2…IN-1) with one less than the number of oscillators (N-1).

Model object
As a model object we have chosen a neural network composed of three thermally coupled VO2-oscillators where each oscillator has the scheme of a relaxation oscillator.Our choice is conditioned by the fact that we have done some research in thermal coupling [11], [17] and its modeling, however, the coupling may be an electric one (capacitive or resistive [15]).It is known that an electric switching effect is observed in VO2 film based structures that is conditioned by a phase metal-insulator transition (MIT) at the moment when the temperature reaches Tt 340 K because of Joule heating by the passing current Isw [18].This gives high-impedance (OFF) and low-impedance (ON) branches on I-V characteristics with threshold voltages (OFFON) Uth  5 V and holding voltages (ONOFF) Uh  1.5 V (see Fig. 3a).Both branches of I-V characteristics are reasonably well approximated by fsw curve consisting of two linearized regions with dynamic resistance Roff 9.1 k и Ron 615 : where Ubv  0.82 V is bias voltage of a low-impedance region, and State is a switch state.One of three presented in Fig. 1 topologies may be realized depending on coupling strength magnitudes .At non-zero ≠0 the topology is "all-to-all" (Fig. 1c), at 2,3=3,2=0 the topology is "star" (Fig. 1b), and at 2,3=3,2=1,3=3,1=0 the scheme turns into a two-oscillator one (Fig. 1a).The control parameters here are source currents Ip1, Ip2, Ip3, their variation leads to alteration of the fundamental oscillation frequency F 0 of oscillators.Variations of each oscillator are described by the equation of Kirchoff's law: where Ui(t) is output voltage taken from the capacitor (C=100 nF), Isw(i)(t)=fsw(Ui(t)-Uin) is current passing through a switch, determined by I-V characteristics (1), Ip(i) is i-th oscillator supply current, respectively, Uin is amplitude of switch internal noise, i is oscillator's number.
Thermal interaction between the i-th VO2-oscillator and the neighbor ones ( (i+)clockwise and (i-)counterclockwise of the scheme in Fig. 3b) is realized according to the rule: If the states of oscillators State(i+) and State(i-) are on the OFF branch of I-V characteristics then the threshold voltage of the i-th VO2-oscillator does not change: Uth(i) = Uth.Rule (3) is the same for all oscillators (with regard to cyclic permutation).
Oscillograms of oscillations with ~250 000 points and time interval t=10µs were simulated using (1-3).After that the oscillograms were automatically processed, synchronization order was determined and cross-sections of oscillator synchronization areas were built.
The switch parameters did not change in numerical simulation of the results, but current intensities Ip, coupling strength  and noise amplitude Uin varied.

Method of synchronization order definition
The problem of finding the high-order synchronization value determined by the ratio of integers k1:k2:k3:..kN (see section 2.1) may be solved in several ways.For example, by direct analysis of all oscillations spectra or by searching the synchronization order of each pair of oscillators based on the method which we suggested in [11].
It also should be noted that the algorithm of pair synchronization definition is based on the search of current oscillation peaks Isw synchronous in time [11], in which case we used the threshold value of synchronization efficiency ηth=90%, and the maximum value of k did not exceed kmax ≤100.

Results
The results of synchronization areas modeling for a two-oscillator scheme (see Fig. 1a) are given in Fig. 4. Control parameters are oscillator feed currents Ip1, Ip2, noise and coupling strength values are Uin=20mV and =0.2V.It can be seen that there is a whole family of synchronization areas that are called Arnold tongues [9].The number of possible variants of synchronous states Ns (information capacity of associative memory) in which the system may exist while the control parameters are varied, is Ns =9.Based on the method presented in section 2.1, to plot vectors E it is necessary to set the neural network at such initial point of control parameters (S) that is located outside the system's synchronization area.At non-synchronous state a system may potentially reach one of the various synchronization states Ns, the transitions are denoted as vectors E1...ENS in the figure.The dimension of the stored vectors in this case is 2, and the coordinates determine current shifts E1=(Ip1, Ip2) with respect to the initial point S.
The problem here is that the synchronization areas are long-ranged (of Arnold's tongues shape) therefore there is a wide range of stored patterns coordinates which bring the system into a certain synchronous state.The solution lies in narrowing the dispersion of stored pattern coordinates by using vectors E'1=(I1, I2…IN-1) of dimension less by one than the number of oscillators (N-1), in this case E'1=(I1).In practice this means that we fix the current for one oscillator and vary the currents for the others (see Fig. 4, dashed line Ip2=const).Thus we eliminate the ambiguity of synchronization definition by one of vector coordinates and the areas of possible synchronization are narrowed.Fig. 5 shows cross-sections of synchronization areas for a system consisting of three oscillators ("star" and "all-to-all", see Fig. 1 b-c) at fixed current on the first oscillator Ip1=const, and parameters Uin=20 mV and =0.2 V that are similar to a two-oscillator scheme.
It can be seen that the synchronization areas are separate isolated regions that are suitable for setting vectors E' of dimension 2. In this case, with all other things being equal, the capacity of associative memory Ns depends on the topology and is Ns=28 for a "star" connection and Ns =21 for an "all-to-all" connection.The area shape also depends on the topology.
When comparing the values for two-and three-oscillator schemes with the same parameters including the topology we may propose a general rule stating that with the increase of the number of interacting oscillators the information capacity of associative memory Ns increases.Yet, this is evident as the number of freedom degree increases at determining the synchronization value k1:k2:k3:..kN.
Nevertheless, as we show below, at certain parameters there are some exceptions from the general rule.Fig. 6 shows the curve Ns versus Δ at three different configurations of a neural network at the constant noise level Uin=20 mV.The existence of the main maximum Ns is evident at some optimal value Δopt, in this case this value is roughly the same as Δopt ~0.1 V for all configurations and does not depend on the oscillator number.The existence of the curve maximum Ns(Δ) reduplicates our results obtained in [10] for a two-oscillator scheme.Inalterability of Δopt for different N, with all other parameters being equal, might be explained by the fact that with the increase of the number of freedom degree for synchronization order k1:k2:k3:..kN the value of Ns has the tendency to grow.
We also should note the general tendency for Ns to decrease when the coupling strength Δ grows above Δopt, and at its large values the system tends to the lowest possible Ns=1 with synchronization value 1:1:1.It is related to the fact that with the increase of Δ the surface of certain synchronization areas increases.Neighboring areas merge; in this case the synchronization order of the resulting area predominantly consists of lower harmonic numbers.As the dimension of the control parameters is limited such growth of synchronization area surfaces irrevocably results in decrease of their number and value of Ns.The insertions in Fig. 6 show the evolution of synchronization areas and demonstrate the effect of their merging with Δ growth.Besides we should note the existence of local maximums at Δ>Δopt.In its turn this is related to the fact that in the presence of noise in the neural network the increase of Δ may result in development of new synchronization areas at the control parameters values that previously corresponded to the non-synchronous state of the system.Therefore in a general case the curve Ns(Δ) may have a complicated shape with several maximums as we can see in Fig. 6.
In addition, the initial sharp growth of all three curves at the plot should be noted when Δ increases from 0 to Δopt.The latter is due to synchronization effect degradation at Δ0.
When comparing the curves it may be noted that the increase of the oscillator number in the network leads to the increase of the maximum value of Ns in the system, this does not contradict the rule suggested above.For example, Ns_max=17 is for two oscillators, for three-oscillator schemes Ns_max=28 and Ns_max=45 are for the "all-to-all" and "star" schemes, respectively.At the same time at certain values of coupling strength (for example, at Δ=1.1 V) the value of Ns for a two-oscillator scheme may be even higher.Also the regularity that the "star" topology has higher Ns than the topology of "all-to-all" is observed.All of these things mean that the increase of the coupling number may contribute to the effect of the system desynchronization and decrease of Ns, it seems that oscillators prevent each other from synchronization.Fig. 7 shows the curve of Ns vs. noise level in the system Uin at the same coupling strength Δ=0.2 V for three configurations of an oscillator neural network.The general trend for the decrease of Ns at the noise amplitude increase is due to the decrease of the surface of synchronous areas which eventually disappear (see the insertions in Fig. 7).It also should be noted that the general rule is observed; stating that the value of Ns for a tree-oscillator configuration "star" is higher than that for a two-oscillator one.The shapes of curves Ns(Uin) are similar which indicated that the physics of noise effect on the network is similar and does not depend on the number of oscillators.
Comparing the above curves Ns(Δ) and Ns(Uin), it may be seen that the maximum value of associative memory capacity Ns_max in our models may reach Ns_max ~450, at optimal coupling strength value Δ=Δopt and lowered noise Uin=10μV the capacity increases to Ns_max ~ 650.
We have assumed that the upper limit for Ns_max is in proportion to the exponential function: where kmax is the maximal number of the harmonic participating in synchronization; and N is the number of oscillators.For example, Fig 8 shows the calculated curve of Ns_max for two coupled oscillators (N=2), calculated according to the map of possible synchronization states distribution (Fig. 8b) and its approximation by Ns_max=0.6(kmax) 2 .
Here the red squares visualize the allowed states of synchronization k1:k2, whose number is within the area (k1≤kmax and k2≤kmax, see the example for kmax=16 in Fig. 8b) and corresponds to Ns_max.The white squares show the states that are not included into the calculation as they form simple proportions with one of the states (for example, all states when k1=k2 is the analogue the state 1:1, and states 2:4 and 4:8 are the analogues to the states 1:2 and so on.)Fig 8a shows the calculated curve Ns_max(kmax) for a three-oscillator scheme and its approximation by Ns_max=0.85(kmax) 3 .It can be seen that at the same kmax the number of synchronization variants for N=3 is much larger than for N=2, which indicates the increase of the states number with the increase of the number of freedom degree for synchronization order and is observed in the shown above curves Ns(Δ) and Ns(Uin).Below we give the example of pattern storage for the scheme "star" at Δ=0.2 V and Uin=20 mV (see Fig. 5а).
Let us assume that we have to store three patterns with coordinates E'1 = (-264 μA, -151 μA), E'2 = (-91 μA, -163 μA) and E'3 = (-88 μA, -290 μA).Than we should set the initial point S at the coordinate S = (847 μA, 921 μA), where synchronous state does not exist.The number of coordinates in each pattern is one less than the number of oscillators and equals 2, this is necessary to narrow the area of possible synchronization and to lower the recognition ambiguity, as we have explained earlier.
When test vector T=E'1 is supplied the neural network is transformed into a synchronous state with the synchronization order 1:2:1.When test vector T=E'2 is supplied the neural network is transformed into a synchronous state with the synchronization order 1:1:1.When test vector T=E'3 is supplied the neural network is transformed into a synchronous state with the synchronization order 2:2:3.
When the coordinates of test vector T deviate from the values of the stored patterns E', within ~1 μA at least, vector T is also positively recognized.
Thus, we have performed storage and recognition of three various patterns, although the capacity of this system is considerably higher and enables storing up to 28 patterns simultaneously.

Conclusion
A new method of pattern storage and recognition in an impulse oscillator neural network based on the high-order synchronization effect is presented using computational modeling of two-and three-oscillator schemes with thermally coupled VO2-switches.In comparison with the proposed earlier single-frequency FSK method [8] this method provides significantly higher information capacity of associative memory states Ns, where each state of the system is characterized by synchronization order k1:k2:k3:..kN.
A general rule is suggested stating that Ns increases with the increase of the number of interacting oscillators.The modeling demonstrates achievement of Ns of several orders: Ns~650 for a three-oscillator scheme and Ns~260 for a two-oscillator scheme.It is shown that the theoretical limit for Ns depends on the number of oscillators and can be estimated as an exponential function.
Several regularities of functional (capacitive) characteristics of such ONN have been obtained; in particular, the existence of an optimal coupling strength between oscillators has been revealed, when the number of synchronous states is maximal.A general tendency for the information capacity decrease with the increase of coupling strength and switches inner noise amplitude is also shown.
An algorithm of patterns E1...ENS storage and test pattern T recognition is proposed.In this algorithm it is better to use the number of coordinates one less than the number of oscillators (N-1) that is necessary to narrow the area of possible synchronization and to lower the recognition ambiguity.
Although the research has been performed on a certain model object (VO2 thermally coupled relaxation oscillators), the demonstrated method of associative memory realization is sufficiently general and the fundamental character of the obtained regularities may be the subject of further research of ONN of various mechanisms and oscillators coupling topology.

Fig. 2 .
Fig.2.a) Example of oscillation spectra of three electric oscillators at synchronization order k1:k2:k3=3:6:4, b) schematic illustration of the synchronization areas for a threeoscillator scheme, c) example of vector and image association, where Ip is current of a power source, F 0 is first harmonic, k is the harmonic number at the synchronization frequency Fs.

Fig. 3 .
Fig. 3. Experimental and model I-V characteristics of VO2-switch (a); a model scheme of a neural network based on three oscillators circuits with VO2-switches interacting via thermal coupling (b).

Fig. 4 .
Fig. 4. Example of synchronization areas for a two-oscillator scheme.The arrows show sampled vectors E and E' in regard to the initial point S.

Fig. 5 .
Fig. 5. Example of synchronization areas for three-oscillator schemes "star" (a) and "allto-all" (b).The arrows show selective vectors E' in regard to the initial point S.

Fig. 6 .
Fig.6.Curves of associative memory capacity Ns vs. coupling strength value between oscillators Δ at various configurations of oscillator neural network and constant noise level Uin=20 mV.The insertions show the evolution of synchronization areas and demonstrate the effect of their merging with Δ growth.

Fig. 7 .
Fig. 7. Curve of associative memory capacity Ns vs. noise level Uin at various configurations of an oscillator neural network at the same coupling strength Δ = 0.2 V.

Fig. 8 .
Fig.8.a) Curves of maximal information capacity of associative memory Ns_max and their power approximations for two (red curve) and three (blue curve) oscillator schemes vs. the maximum number of harmonic kmax; b) distribution of allowed states of synchronization (red squares) and forbidden ones (white squares) for N=2, the dashed line restricts the area for kmax=16.