#### 2.1. Oscillator Circuit and Method of ONN Organization

The basic element in the studied ONN is an oscillator implemented on the circuit of a relaxation generator (

Figure 1) based on a VO

_{2}-switch. We described the process of fabrication and electric I–V characteristics of an electric VO

_{2}-switch in detail in [

19]. I-V characteristics are well approximated by a piecewise linear function (see

Appendix A.1) that has two conduction states (low-resistance and high-resistance) and a region of negative differential resistance (NDR). These switches may be used in the circuit of a relaxation generator with power supply

I_{p} holding the operation point in the NDR region of the I-V characteristic and with capacity

C, connected to the switch in parallel. In addition, a source of noise

U_{n} models the circuit’s interior or exterior noises, such as current noise of a switch [

34]. The oscillator’s output signal is current

I_{sw}(

t) flowing through the VO

_{2}-switch, which is used to calculate the synchronization level of two oscillators (see

Section 2.3). The current signal directly determines the effect of thermal coupling inside the network.

We applied thermal coupling to connect oscillators in a network. The presence of the thermal coupling mechanism between two VO

_{2} oscillators was convincingly demonstrated by us in the experiment [

19], and it is based on the mutual thermal effect of switches due to their Joule heating and the dependence of the threshold voltage

U_{th} on temperature. In the pre-threshold mode, when the VO

_{2} temperature of the switch channel is close to the MIT temperature

T_{th} ~ 340 K [

23], the external thermal influence can “push” it to switch, which is equivalent to lowering the threshold voltage by the value of

s, the thermal interaction force. The switching causes an even higher temperature rise in the switch, which leads to the appearance of a temperature pulse in the surrounding space, which propagates as a temperature wave. In the oscillator circuit, the thermal effect of the switches on each other occur in the mode of repetitive pulses initiated by self-oscillations of oscillator currents, which ultimately leads to their synchronization [

19]. The value of

s can be controlled experimentally, for example, by varying the distances between the switches or by varying the parameters of the external circuit [

19].

The choice of this coupling type is determined by the simplicity of a computing model when oscillators are electrically separated from each other, unlike in capacitive or resistive couplings [

21,

22,

25,

28]. An example of an oscillators’ interaction via thermal coupling of VO

_{2} switches is shown in

Figure 1. A detailed presentation of the mathematical model of thermally coupled relaxation oscillators is given in

Appendix A.1.

The heat assisted model we used completely describes the experimental data previously observed [

19,

35,

36]. Indeed, the speed of propagation of a thermal wave can be taken into account, but this is not in the scope of the current paper because we operated at low oscillation frequencies where the time delay is insignificant. We considered the radius of thermal interaction

R_{TC} in [

19], therefore, in this model we limited ourselves to interaction with the nearest oscillators. Consideration of more complex cases can be done in future publications.

We used the concept of one-way thermal coupling of oscillators in the numerical model. This can be physically implemented when a resistance heater is used as a connecting element in one of the circuits instead of a switch [

19].

For numerical modeling we used the following parameters as I-V characteristics (

U_{th} = 5 V,

U_{h} = 1.5 V,

U_{bv} = 0.82 V,

R_{off} = 9.1 kΩ,

R_{on} = 615 Ω). In this circuit, capacity is a constant parameter

C = 100 nF, and its value significantly determines the frequency range of oscillator operation [

37] and its natural frequency

F_{0}. In our case, the frequency range was 165 Hz ≤

F_{0} ≤ 1266 Hz at the range of feeding currents 550 µA ≤

I_{p} ≤ 1061 µA.

#### 2.2. ONN Structure

The studied ONN consists of an input pattern, which is transferred as a 3 × 3 matrix on the layer of processing neurons consisting of 9 oscillators, and of an output layer with only one oscillator (output neuron) No.10 (see

Figure 2).

The magnitude of the effect of the

i-th oscillator on the

j-th oscillator is determined by coupling strength

s_{i,j}, that is set by the following matrix:

The layer of processing neurons consists of a 3 × 3 oscillator matrix connected by similar couplings (

s_{i,j} =

s_{j,i} =

s_{m}, where

i,

j are the numbers of neighboring oscillators). Neighboring oscillators are only connected by horizontal and vertical lines. So, the central oscillator No.5 has four couplings in the matrix, the corner oscillators have two couplings, and the oscillators in the center of the edges have three couplings (with the central oscillator and two corner ones) and they all unidirectionally affect oscillator No.10 (output neuron) in the output layer (

s_{i,10} =

s_{o} and

s_{10,i} = 0, where

i = 1 … 9). Importantly, there is the reference oscillator No.0 in the circuit and the synchronization order of all other oscillators is measured against this oscillator. Oscillator No.0 (

Figure 2) unidirectionally affects all other oscillators (

s_{0,i} =

s_{r}, and

s_{i,0} = 0, where

i = 1 … 10).

The input pattern (see

Figure 3) is transferred to the processing layer by selection of the feeding currents of the oscillator matrix:

where

i = 1 … 9,

x_{i} are the coordinates of the input vector X = (

x_{1}, …,

x_{9}), that correspond to white (

x_{i} = 0) and blue (

x_{i} = 1) colors of pattern squares.

Transfer of the pattern to the intermediate layer causes the change in the oscillators’ feeding currents, and in turn, it leads to the change in synchronization state for all oscillators (No.1–10). The synchronization order SHR

_{0,10} between the reference oscillator No.0 and the oscillator in the output layer No.10 serves as the control value. The values of the feeding currents for these oscillators are fixed and may differ from currents in the matrix, therefore they are indicated as

I_{0} and

I_{10} (see

Figure 3).

#### 2.3. Method of Synchronization Order Definition

In this section, we present the method of calculating a family of metrics [

19] to determine the high order synchronization of two oscillators. A family of metrics has two basic parameters: the ratio of subharmonics (SHR) and synchronization effectiveness

η. In a general case, it is possible to use the concept of subharmonics ratio between oscillators

i and

j, which is defined [

18] as a simple fraction

where

k_{i} and

k_{j} are harmonics order of oscillators at the common frequency of their synchronization

F_{s}.

In other words, Equation (3) uses spectral approaches to describe synchronization of the higher order

k_{i}:

k_{j}, when the following relation is observed:

where

F_{i}^{0},

F_{j}^{0} are frequencies of main harmonics.

As an example,

Figure 4 shows the qualitative spectra of two oscillators at SHR

_{i,j} = 2:7

In addition,

Figure 4 shows the total synchronization frequency

F_{s} and subharmonics numbers at this frequency

k_{i} = 2 and

k_{j} = 7. Usage of signal spectra for estimating the magnitude of SHR

_{i,j} is not effective because the signals might not have a strictly periodic sequence and when noise is added to the system, the spectral lines broaden. Below, we will describe the mathematical procedure that estimates the value of SHR

_{i,j} without the use of spectral characteristics but using a current signal and array LE. Array LE stores information on the position of the current pulse leading edges (see

Figure 5).

Figure 6 shows arrays LE[

i] and LE[

j] for two oscillators that correspond to the same ratio of the basic frequencies as shown in the spectra in

Figure 4. The distance between two nearest phase-locked pulses is denoted as

T^{z}_{s}—the period of synchronization (where

z is a conditional number of periods

T_{s}).

Therefore, SHR

_{i,j} may be estimated using a phase-locking method:

where

M_{i} and

M_{j} are the number of signal periods falling into the synchronization periods

T^{z}_{s} of two oscillators. Formula (5) can be easily obtained from Formula (3) taking into account Formula (4) and the following ratio (

T_{s} =

M_{i}·

T_{i}^{0} =

M_{j}·

T_{j}^{0},

F_{i}^{0} = 1/

T_{i}^{0},

F_{j}^{0} = 1

/T_{j}^{0}).

In general, especially when a system behaves erratically, synchronization periods differ and spread in T

^{z}_{s} ≠ T

^{z}^{+1}_{s} and the values of

M_{i} and

M_{j} may change within one oscillogram (see

Figure 7).

Various values of synchronizations SHR

_{i,j} may occur within one oscillogram. To determine which SHR

_{i,j} value prevails, it is necessary to find the occurrence probabilities

P(

M_{j}:

M_{i}) for each pair (

M_{i}:

M_{j}) that is present in the whole oscillogram and to select the pair with the maximum value of

P = P_{max}(

M_{j}:

M_{i}). Then, the final value of SHR

_{i,j} will be written as:

To find the probabilities

P(

M_{j}:

M_{i}), we can count how many times

NP(

M_{j}:

M_{i}) the given pair appeared within the whole oscillogram of the oscillator

i, multiply by the number of periods in it (

M_{i}) and divide by the total number of all oscillations periods in the given signal (

N_{j}). Thus, for

P(

M_{j}:

M_{i}) we obtain:

where

N_{i} is the total number of periods in the oscillogram of oscillator

i.

It is convenient to present the probabilities

P(

M_{j}:

M_{i}) as a histogram where the values are positioned in the descending order of the magnitude

P. For example, for the oscillogram section in

Figure 7 the following histogram can be built. The histogram in

Figure 8 is calculated by Formula (7), when the pairs occur the following number of times,

NP(2:7) = 2,

NP(2:9) = 1,

NP(2:5) = 1, and the total number of periods is

N_{i} = 28 (in real calculations,

N_{i} was in the range of 1000–3000 for greater accuracy, see

Appendix A.2).

For SHR

_{i,j}, the parameter of synchronization effectiveness

η is defined as the maximum probability

P_{max}(

M_{j}:

M_{i}):

Therefore, we define the synchronization calculated above as SHR_{i,j} = 2:7 with effectiveness of η = 50%.

The family of metrics (SHR

_{i,j},

η) allows sufficient determination of the synchronization state of two oscillators and calculation of the distance between the states, i.e., the difference between the synchronization degree. This property allows the use of metrics in an oscillator neural network training, pattern recognition systems and artificial intelligence [

8,

9,

10,

11].

Depending on the task, for example, the network training for data coding and pattern recognition, the problem of the presence or absence of synchronization SHR

_{i,j} can be solved by formally setting the synchronization effectiveness threshold

η_{th}, so

In the majority of cases, we set

η_{th} = 90%, meaning the signals are synchronized if 90% of their durability have a certain synchronization pattern. For the network training, this parameter can be chosen within a selected range, and it is one of the important parameters of the network adjustment [

33].

The main technical problem we faced, was the problem of defining the synchronization order between the reference oscillator No.0 and the oscillator of the output layer No.10 characterized by the value SHR

_{0,10}:

The same value of SHR_{0,10} can be expressed in several ways: as a ratio, a simple fraction or a real number, for example, SHR_{0,10} = 10:3 = 10/3 = 3.33. Later, we will use this property to present the results more vividly.

Parameter SHR_{0,10} has the properties of an output neuron while the reference neuron may be considered as a master generator to which we calculate synchronization of other network neurons.

Two parameters SHR and η are used as the main metrics for evaluating the degree of two oscillators’ synchronization and are applied in the algorithm of ONN training.

The current oscillograms

I_{sw}(

t) of oscillators No.0–10 were calculated simultaneously and contained ~250,000 points with the time interval Δ

t = 10µs (see

Appendix A.1). Then, the oscillograms were automatically processed.

The switch parameters were unchanged in numerical simulation (see

Section 2.1), while current intensities

I_{p_i} (

I_{ON},

I_{OFF},

I_{0,} I_{10}), coupling strength constants

s (

s_{r},

s_{m},

s_{o}), noise amplitude

U_{n} and

η_{th} varied.

#### 2.4. Pattern Classifier Implementation and Problem Definition

A “black-and white” pattern was used as an input test pattern presented as a 3 × 3 matrix (without gradation of gray color, 3 by 3 pixels). The form of the pattern can be unequivocally defined by the input vector X

_{n} = (

x_{1}, …,

x_{9}) where each cell may take the value

x_{i} = 0 (white color) or

x_{i} = 1 (blue color), and

n is the number of the vector equal to the decimal value of the coordinates presented as a binary sequence. For example,

Figure 2 and

Figure 3 show an input pattern that corresponds to the vector X

_{489} = (1,1,1,1,0,1,0,0,1). The total number of patterns (vectors)

n in the input layer matrix is 2

^{9} = 512 (X

_{0} … X

_{511}). Presuming that the pattern processing layer together with the output layer has certain symmetry, a set of 512 vectors X

_{n} may be divided into 102 classes C

_{j}, where

j is the number of classes (

j =1 … 102) (see

Figure 9). The complete list of classes and their elements is described in

Supplementary Materials (Data1.txt).

The principle of figure distribution into classes is as follows: each class C_{j} consists of a lot of figures (vectors) that have the same number of blue (white) cells and rotation-reflection axial symmetry of the 4th order (symmetry at rotation by 90°).

Mirror-rotation axial symmetry of the 4th order supposes an association of all figures in a separate class in the mirror operation in relation to the central columns (horizontally and vertically) and also at rotation by 90° (see the example of figures transformation for class C

_{4} in

Figure 10).

Figures from one class impose the same effect on the neural network and have the same output value SHR_{0,10}. Distribution into classes allows us to find figures that at any current values (I_{ON}, I_{OFF}, I_{0}, I_{10}) and coupling strength constants (s_{r}, s_{o}, s_{m}), noise amplitude U_{n} and η_{th}, will have the same values of synchronization effectiveness η and SHR_{0,10} within one class of figures as a result of neural network symmetry. The initial distribution of all 512 figures into classes allows us to recognize not one specific figure, but a class (out of 102 possible ones) into which this figure is classified. For example, for class C_{5} the equality SHR_{0,10}(X_{5}) = SHR_{0,10}(X_{260}) = SHR_{0,10}(X_{320}) = SHR_{0,10}(X_{65}) will be observed.

Such classification reduces the number of possible variants to sort as we may send only 102 figures to the input layer, one figure per each class.

The problems this neural network is able to solve may be divided into several variants:

In fact, problem I is a subcase of problem II at P = 1.

The principle of solving problem II is shown in

Figure 11. Here

P = 3, and set

**C** = {C

_{1}, C

_{4}, C

_{5}}, in this case the corresponding set is

**SHR** = {16:15, 13:10, 14:13}. Therefore, unambiguous recognition of figures belonging to three different classes takes place. Synchronization is not realized for all other classes and

η <

η_{th}.

As there is a huge variety of set **C** variants, this problem can be reduced to the search of possible solutions with P > 1 and to the determination of the maximum value of P.

- III.
The third variant of the problem corresponds to a fully trained network when it solves problem II for all possible input classes C_{j}, when P = 102.

In this problem, each class out of 102 possible variants corresponds to its unique value of synchronization order SHR_{0,10}. Therefore, the set of input classes **C** = {C_{1}, C_{2} … C_{102}} transfers into a set of synchronous states **SHR** = {SHR^{(1)}_{0,10}, SHR^{(2)}_{0,10} … SHR^{(102)}_{0,10}}, where all the elements have non-duplicate values.

Problems I-III have an increasing degree of complexity and are subcases of problem II with different parameter

P. Problem I is the simplest one and it may be solved by a common neural network with one bistable output neuron (bistability means the presence or absence of synchronization), although two variants of answers in neural networks are often given by using two output neurons [

38].

The output neuron in problems II and III have multilevel properties and differ from a common bistable neuron. All three problems can be applied to the circuit shown in

Figure 2. The ONN circuit has only one output neuron, nevertheless, the multilevel high order synchronization SHR

_{0,10} allows input pattern classification into

P classes within set

**C**. This is the most striking difference between the described neural network and variants presented in the literature. This increases the net data throughput of a single neuron and enables us to create multilevel output cascades of neural networks with high functionality.

#### 2.5. Technique for ONN Training

To solve problems I–III, network training is required. As ONN with a high order synchronization effect has not been studied before, there are no established methods for network training. One of the ways is to use a simulated annealing algorithm [

39] for the network parameters selection: current (

I_{ON},

I_{OFF},

I_{0,} I_{10}), coupling strength (

s_{r},

s_{o},

s_{m}), noise amplitude

U_{n} and the synchronization effectiveness threshold

η_{th}. The algorithm’s main point is the random searching of the problem II solution with the maximum value of

P at some initial interval of parameters, followed by the narrowing of these intervals.

Determination of step size is required for direct parameter searching. In our numerical experiment, the current range was determined by the generation of oscillation in a single oscillator. For the oscillator circuit described in

Section 2.1, the generation of oscillation was within the feeding current range of 550 µA ≤

I_{p} ≤ 1061 µA. Therefore, the currents (

I_{ON},

I_{OFF},

I_{0,} I_{10}) varied in this range. We determined the variation steps as δ

I_{p_i} = 1 µA. So, there were 512 current steps.

For coupling strength

s, the variation range was determined by the threshold values of I–V characteristics of a switch. The condition, that is, the integral thermal action on the neuron

s_{Σ} should not reduce the threshold voltage of the I–V characteristic of a switch

U_{th} below the holding voltage

U_{h}, must be met:

On the basis of Formula (13), for the values of threshold voltages (

U_{th} = 5 V,

U_{h} = 1.5 V) and coupling configurations (see

Figure 2), the limits of the coupling strength variations are subject to the following conditions: (

s_{r}+

s_{o}·9) < 3.5 V and (

s_{r}+

s_{m}·4) < 3.5 V. This condition is met with the following ranges that we have chosen:

s_{r} = 0 ÷ 0.2 V,

s_{m} = 0 ÷ 0.5 V,

s_{o} = 0 ÷ 0.3 V. The variation step was chosen as 0.1% of the range magnitude.

The range 20 µV ≤ U_{n} ≤ 900 µV with the number of steps equal to 12 was chosen for the noise amplitude.

For the synchronization effectiveness threshold η_{th}, the variation range was 10 % ≤ η_{th} < 100%, with the number of steps equal to 25 and minimal spacing δη_{th} = 0.1%. Parameter η_{th} does not belong to the network parameters but rather to the parameters of the algorithm of synchronization order SHR_{0,10} calculation. The value η_{th} strongly affects the results of synchronization and the results of pattern recognition because it is a conditionally chosen characteristic. Identifying its optimal value for the recognition problems is an important step in network tuning and training.

Having a lot of network parameters, each parameter’s variant should be calculated in 102 classes together with oscillograms and synchronization values SHR_{0,10} at each stage. A full, direct search of all parameters’ variants would take a lot of time and computational resources. Therefore, we fixed the values U_{n} and η_{th}, and varied currents (I_{ON}, I_{OFF}, I_{0}, I_{10}) and couplings strength (s_{r}, s_{o}, s_{m}). The algorithm for searching for the solution of problem II with the maximum value P included the following steps:

Preparation step: Setting of constant values of U_{n} and η_{th}

- Step 1:
Random searching of parameters (I_{ON}, I_{OFF}, I_{0}, I_{10}, s_{r}, s_{o}, s_{m}) in the maximal range of their variations and finding the values meeting the maximum value P. The number of searching attempts is 1000.

- Step 2:
Narrowing of the parameters ranges by 5 times with their symmetric distribution in relation to the results of the previous step. The number of searching attempts is 1000.

- Step 3:
Narrowing of the parameters ranges by 25 times with their symmetric distribution in relation to the results of the previous step. The number of searching attempts is 1000.

Noise amplitude

U_{n} is kept fixed because this is the parameter that is not often controlled in the experiment. It is determined by external and internal circuit noises, and we considered its effect individually in more detail. We used

U_{n} = 80 µV, which is close to the experimentally observed value [

19].

The threshold value of synchronization effectiveness η_{th} is fixed because it is the main parameter in the synchronization algorithm, and we considered its effect individually in more detail. For the pattern recognition experiments, we used a workstation (Intel Xeon quad core processor, Albuquerque, NM, USA, 4 × 2 GHz, 8GB RAM) running 64-bit Windows Server 2008. CPU time for a single run on one core for the direct search procedure with 1000 search attempts took ~5 h.