Randomized Modality Mixing with Patchwise RBF Networks for Robust Multimodal Pain Recognition ^†

Abstract

Pain recognition based on multimodal physiological signals remains a challenge, not only because of the limited training data, but also due to the varying responses of individuals. In this article, we present a randomized modality mixing technique (Modmix) for multimodal data augmentation and a patchwise radial basis function (RBF) network designed to improve robustness in limited and highly heterogeneous data. Modmix generates new samples by randomly swapping modalities between existing data points, creating new data in a very simple but effective way. The RBF patch network divides the input into randomly selected, overlapping patches that capture local similarities between modalities. Each patch network is trained end-to-end using stochastic gradient descent. Moreover, the model’s performance is further improved by using multiple independently trained networks and combining them into a single decision. Experiments with the two different pain datasets X-ITE and BioVid were performed under limited training data conditions, where only approximately 30% of the original datasets were used for training. With both datasets the RBF patch network achieved significant improvements for a subset of subjects, resulting in a similar or even slightly better mean accuracy compared to competing related models such as random forest and support vector machine.

1. Introduction

Automatic pain recognition is required in situations where patients cannot communicate their pain verbally in the usual way. As well as with many other fields, machine learning is delivering promising results in pain recognition. In and of itself, machine learning can be viewed as a fusion of statistical inference and function approximation. Both bring their own challenges, summarized in the so-called bias–variance trade-off. Here, variance measures how well a model generalizes from finite data, while bias measures how well it can represent the target function. In the area of deep learning where models can represent arbitrary functions, bias plays a rather minor role, and the main focus is on variance, which usually arises from insufficient data, noise, or unrepresentative distributions. Interestingly, areas such as pain recognition suffer primarily from a lack of data, as the corresponding handcrafted datasets were designed by experts who know which signals to measure, how to measure them, and, above all, where to look for the required features in the signal. Such high-quality datasets are therefore extremely expensive and possibly even ethically questionable.

For this reason, among others, researchers have started to use data augmentation techniques to generate additional samples from existing ones [1]. However, while standard techniques such as geometric transformations or noise addition offer limited novelty, mixup augmentation [2] produces new samples through convex combinations of data samples and labels. In [3], we propose a mixup variant to address multimodal pain detection where feature vectors are composed of self-contained modalities originating from different signal sources. The relatively high degree of independence between the modalities makes it possible to generate new data by simply exchanging modalities between existing data samples. The introduction of randomness to improve generalization is a well-known and widely used approach in machine learning, for example, in dropout regularization, stochastic gradient descent learning, and especially in random forests where randomization is applied at two levels: at the dataset level by sampling subsets for each decision tree and at the feature level by using only a sample of features for each tree. By averaging the predictions of these simple trees with consequently high variance, the ensemble achieves robust variance reduction.

Likewise, we introduce a radial basis function (RBF) network that operates at localized random patches of the feature vector. Each patch represents a predefined, randomly selected multi-set of features and provides a localized view of the high-dimensional data to capture different structures distributed across different modalities. We then combine multiple different RBF patch networks and take the average output as the final decision, with each network trained on somewhat different data augmented with Modmix. Unlike conventional neurons, RBF neurons measure similarity based on the distance between points and simultaneously represent an inner product of a higher-dimensional space in which linear decision boundaries can be found. This trick is also used in support vector machines (SVMs). However, SVMs are far less flexible and cannot be easily adapted to a patch-based approach such as neural networks. In addition, they often require a large number of support vectors to represent the decision functions. Therefore, our RBF patch network combines the compactness and flexibility of neural networks with the advantages of kernel-based optimization.

The rest of this article is structured as follows: Section 2 provides a concise overview of previous research on pain recognition and its machine learning approaches. We then present our proposed RBF patch network and Modmix data augmentation strategy in detail in Section 3. The multimodal pain datasets X-ITE and BioVid, which we use in our experiments, are introduced in Section 5. The results and discussion of the experiments are presented in Section 6. We then conclude this article in Section 7 with a summary of our findings and possible future research directions.

2. Related Work

SVMs with radial basis function (RBF) kernels and random forests have already proven to be powerful models in the field of pain recognition based on facial expressions, electromyography, or physiological signals. Both have been widely used in datasets such as UNBC-McMaster [4], where they performed exceptionally well. For a multimodal distribution in which heterogeneous features such as facial action units and skin conductance are combined, random forests, in particular, have been shown to be effective in maintaining robustness to noise and small sample sizes. In this way, these results underscore the importance of localized similarity measures and ensemble averaging for reliable pain detection.

Deep learning and its powerful model have opened up entirely new possibilities for recognizing useful patterns in facial, physiological, and sometimes even brain activity signals [5,6,7]. Recently, radial basis function networks have regained popularity due to their ability to recognize local patterns in input data, which also makes them easier to interpret. These properties can be especially helpful in pain detection, as various studies have already demonstrated [8,9]. The closest study to our work is [10], where RBF networks are used in convolutional layers with patch-like local connections. Just like conventional neural networks, RBF networks are universal approximators. Even so, universal approximation theorems prove expressiveness but make no assumptions about how much data is needed.

Recent transformer-based architectures have shown promising results in capturing temporal dependencies in electrode activity (EDA) and electrocardiogram (ECG) signals [11,12,13]. In [14] a cross-modality framework was introduced, which achieved state-of-the-art performance (87.52% accuracy on BioVid dataset) by employing a two-stage fusion strategy: first extracting intramodal features using transformer blocks for each signal independently and then fusing EDA and ECG representations through a dedicated cross-modal transformer.

On the other hand, availability of sufficient data is a common challenge in machine learning, especially in deep learning. Data augmentation has thus become a crucial part of developing successful machine learning models. Nonetheless, generating representative synthetic data is a highly complex problem in itself, but fortunately, mixup is a very simple method that delivers extremely good results. Over time, other variants of mixup have emerged, including CutMix [15], which is closest to our work. CutMix combines images by cutting out an area from one image and pasting it into another. The corresponding labels are mixed proportionally to the area of the pasted region. CutMix encourages the model to focus on critical parts of the image when making predictions, improving generalization and robustness, especially for tasks such as object detection.

3. Materials and Methods

3.1. RBF Patch Network

The classic architecture of an RBF network corresponds to a feedforward neural network with only one hidden layer and Gaussians (radial basis functions) as activation. The fundamental difference from conventional neural networks is that the Euclidean distance $\parallel x-c\parallel$ from the feature vector x to the weight vector c is used as input for activation $\varphi$ instead of the dot product $c^{T}x$.

$$\mathsf{\Phi }\left(x\right)=exp\left(-\frac{{\parallel x-c\parallel }^2}{2{\sigma }^2}\right).$$

(1)

The vector c can be considered as some center in the input space, where $\parallel x-c\parallel$ defines a spherical activation area of the neuron in which the center c provides maximum activation. The width parameter $\sigma$ determines how sensitive the neuron is to distances from its center c and is usually derived together with c from the data in a pre-training phase using a cluster algorithm or as part of a global learning algorithm such as gradient descent.

Our proposed network follows this classical structure as shown in Figure 1, but instead of connecting each hidden neuron to the entire input vector, we divide the input into randomly sampled, smaller, possibly overlapping mulitsets, which we will call patches. Each random patch can contain features of each modality and is assigned to a fixed neuron. The advantage of this approach is twofold: the dimensionality reduction preserves the meaningfulness of local distances while maintaining activation diversity, and the randomly selected features enables the model to detect localized similarities across modalities. The outputs of all these local detectors are then linearly combined to make the final prediction

$$f\left(x\right)=w_0+\sum _{j=1}^k{w}_j{\mathsf{\Phi }}_j\left(x_{P_j}\right).$$

(2)

3.1.1. Mahalanobis Distance

Apart from losing its meaning as a distance measure in high-dimensional spaces, there is another limiting characteristic of the Euclidean standard distance: it does not take into account correlations or differences in scale between features. This means that features with larger variances can distort the distance in much the same way that outliers can distort the mean. For this reason we use the Mahalanobis distance $D_M$,

$$D_M(x,c)=\sqrt{{(x-c)}^T{\mathsf{\Sigma }}^{-1}(x-c)}$$

(3)

where the data is first transformed into a space where all features are uncorrelated. In this way points that form an elliptical cluster will be transformed into a space where their cluster is spherical, so that the Euclidean distance will not be distorted.

In the case where the features are uncorrelated, the covariance matrix $\mathsf{\Sigma }$ is diagonal and the points already form a spherical cluster. Then $D_M$ is just a normalized version of the Euclidean distance. In other cases consider that the term under the root in (3) is generally an n-ellipsoid. Since $\mathsf{\Sigma }$ is symmetric, it has an eigenvalue decomposition with orthonormal matrix P and diagonal matrix D, so that its inverse ${\mathsf{\Sigma }}^{-1}$ can be represented by

$${\mathsf{\Sigma }}^{-1}={\left(PD{P}^T\right)}^{-1}=P{D}^{-1}{P}^T$$

(4)

Putting (4) into (3) while using $D^{-1}=D^{-\frac{1}{2}}{D}^{-\frac{1}{2}}$ and $\mathcal{S}=D^{-\frac{1}{2}}{P}^T$ leads to

$$\begin{array}{cc}\hfill D_M(x,c)& =\sqrt{{(D^{-\frac{1}{2}}{P}^T(x-c))}^T(D^{-\frac{1}{2}}{P}^T(x-c))}\hfill \\ & =\sqrt{{\left(\mathcal{S}(x-c)\right)}^T\left(\mathcal{S}(x-c)\right)}\hfill \\ & = \parallel \mathcal{S}x-\mathcal{S}c\parallel \hfill \end{array}$$

(5)

In (5) we eventually see that the Euclidean distance is applied on transformed points. Using $D_M$ in (1) we get

$$\mathsf{\Phi }\left(x\right)=exp\left(-\frac{1}{2}{(x-c)}^{\top }{\mathsf{\Sigma }}_j^{-1}(x-c)\right)$$

(6)

3.1.2. Patch Sampling

Patch sampling plays a crucial role in the architecture of the RBF patch network, where the number m of patches determines the number of neurons processing inputs of size k. The elements of the patches are indices of the feature vector with length n, which are sampled uniformly with replacement.

Some features may thus occur in more than one patch, which on the one hand leads to slight correlations between the neurons and on the other hand offers the neurons different perspectives on the same feature. Some features might not occur in any patches, which is definitely the case for $mk<n$. This could be considered as some form of regularization. It is also possible that some features occur several times in the same patch, giving them more weight in the distance calculation. For a given patch the probability of this happening is

$$\mathcal{P}\left(\mathrm{at}\mathrm{least}2\mathrm{same}\mathrm{features}\right)=1-\frac{n(n-1)\cdots (n-k+1)}{n^k}$$

(7)

To avoid excessive feature redundancy within patches one should keep k relatively small.

3.1.3. Relation to RBF-SVMs

Another well-established and widely used model that uses RBF functions is the support vector machine (SVM) with RBF kernels. Both the RBF-SVM and the RBF network have fundamentally different theoretical backgrounds; however, they represent decision functions of nearly identical algebraic form. For the RBF-SVM, the decision function is given by

$$f\left(x\right)=w_0+\sum _{i\in S}{w}_i{y}_i\mathsf{\Phi }(x,x_i),$$

(8)

where $x_i$ are the support vectors, $y_i$ are their corresponding class labels, and $\mathsf{\Phi }$ uses (1) as a kernel. Each model combines nonlinear basis functions linearly to create a linear classifier in a higher-dimensional space, where these basis functions represent corresponding inner products of the inputs.

However, the approach to determine the decision function differs fundamentally, as does the way in which they separate the classes in the input space. The SVM attempts to maximize the boundary between the classes by choosing as a center the subset of the training data that contains only points closest to the decision boundary on each side. These points are called support vectors, and their number can be as large as the dataset in the worst case. In addition, the same width parameter is used in all kernels. The RBF network is much more flexible; its architecture is essentially user-defined, with the number of neurons being a design decision and the centers ultimately being located anywhere in the input space. Furthermore, the width parameters are also typically learned and can differ between neurons. The RBF network is easy to customize and extend, as in this work. This flexibility often results in more compact models that perform similarly or even better than SVMs.

3.1.4. Relation to Random Forests

In this work, we combine several RBF patch networks with each other and take the average of the individual outputs as a decision. This approach has some similarities with other ensemble models, in particular with random forests. A random forest combines multiple decision trees, each trained on a randomly selected subset of the training data. Furthermore, the individual decision trees are created using only a randomly selected subset of features. This two-stage randomization across data and feature dimensions leads to diversity that effectively reduces model variance and improves generalization.

Similarly, the RBF patch network is randomized at the feature level, with randomly selected patches assigned to individual neurons. Each patch is a multi-set supported by a subset of the input features. Technically the RBF patch network creates a weighted average of single outputs based on subsets of the features but has no randomization at the data level like the random forest does. Randomization at the data level is then introduced by training the individual RBF patch networks on a slightly modified version of the dataset. This difference results from the addition of a certain percentage of augmented data to each network. The artificial data is generated using Modmix, which can be considered a kind of third level of randomization, as new data is essentially generated by the random combination of modalities.

3.2. Data Augmentation with Modmix

Randomized modality mixing (Modmix) is a special form of the mixup data augmentation method. To appreciate the idea of mixup we will first give an informal description of the underlying core principle, vicinal risk minimization (VRM).

3.2.1. Vicinal Risk Minimization

Machine learning in general, and neural network learning in particular, is concerned with the problem of estimating an unknown function based on a finite data sample. Suppose the training data comes from $S=\mathcal{X}\times \mathcal{Y}$ with the joint probability distribution p. Here $\mathcal{X}$ represents the input space and $\mathcal{Y}$ represents the label space. Let h be the function represented by a given model and L measure the loss $L\left(f\right(x),y)$ of h when it makes a prediction with $(x,y)\in S$; then the expected loss of f over S is given by

$$R\left(h\right)=\underset{S}{\int \int }p(x,y)L(h\left(x\right),y)dydx.$$

(9)

The expected loss is also referred as the true risk. However since the distribution p is unknown, R is estimated with

$$\hat{R}\left(h\right)=\frac{1}{n}\sum _{(x_i,y_i)\in D}L(h\left(x_i\right),y_i).$$

(10)

where $D={\left\{(x_i,y_i)\right\}}_{i=0}^n$ is a finite sample of training data. $\hat{R}$(h) is accordingly called the empirical risk. The goal is now to choose h such that $\hat{R}$ is minimized. However, since $\hat{R}$ is only an estimate, the performance of h for unseen data highly depends on the quality of $\hat{R}$. And like any sample-based statistic, the quality of $\hat{R}$ increases with the sample size. However, generating new data can be very expensive or even impossible. Instead of using complex methods to generate artificial data, VRM suggests sampling new points from the neighborhood (vicinity) of individual training points $x_i$, assuming that the corresponding label $y_i$ changes only slightly, if at all [16,17]. Let $v\left(x_i\right)$ be the neighborhood of $x_i$; the vicinal risk V is then defined as follows:

$$V\left(h\right)=\frac{1}{n}\sum _{(x_i,y_i)\in D}L\left(\mathbb{E}\left[h\left(x_i\right)\right],y_i\right),\mathrm{where}\mathbb{E}\left[h\left(x_i\right)\right]={\int }_{v\left(x_i\right)}h\left(x\right)p(x\mid v\left(x_i\right))dx.$$

(11)

3.2.2. Mixup

In mixup the vicinity of point $x_i$ is based on a convex combination with another point $x_j$, while the parameter $\lambda$ for this combination is sampled from a symmetric beta distribution

$$v\left(x_i\right)=\{\lambda x_i+(1-\lambda )x_j|x_j\in D,\lambda \sim \mathrm{Beta}(\alpha ,\alpha )\}.$$

(12)

Notice how all points in D can belong to the vicinity $v\left(x_i\right)$ and that the parameter $\alpha$ of the beta distribution determines the probability of the location of the interpolated points. The beta distribution models possible values in the interval $\lambda \in [0,1]$; while the mean ${\mu }_{\lambda }$ is always $\frac{1}{2}$, the variance is

$$\mathrm{Var}\left(\lambda \right)=\frac{1}{4(2\alpha +1)}.$$

(13)

With an increasing $\alpha$, the variance $\mathrm{Var}\left(\lambda \right)$ decreases and the $\lambda$s will cluster increasingly close to ${\mu }_{\lambda }$, placing interpolated points rather in the middle of the line. On the other side for $\alpha =0$ the beta distribution collapses to a discrete Bernoulli distribution with $p=0.5$, which means that mixup will not create new points anymore. However we will see that this is not the case for Modmix. Since mixup creates points between every possible pair of D, regardless of the class, the label of $\hat{x}$ is also a convex combination, calculated with the same $\lambda$ used for $\hat{x}$:

$$\hat{y}=\lambda y_i+(1-\lambda )y_j.$$

(14)

It is interesting to see what kind of figure mixup creates. Let $\mathrm{Conv}\left(D\right)$ be the convex polytope induced by the dataset D; then the set $\mathcal{M}\subset \mathrm{Conv}\left(D\right)$ of all points is

$$\mathcal{M}=\bigcup _{x\in D}v\left(x\right)$$

(15)

In $\mathcal{M}$ all lines are connected and can be imagined as some kind of mesh of $\mathrm{Conv}\left(D\right)$.

3.2.3. Modmix

Let ${\mathbf{x}}^T=(x_1,x_2,\dots x_m)\in {\mathbb{R}}^d$ be a multimodal feature vector, where $x_i\in R^{d_i}$ represents the i-th multidimensional modality and $d=\sum d_i$. In Modmix a new point $\hat{\mathbf{x}}$ is created by randomly selecting two points ${\mathbf{x}}_i,{\mathbf{x}}_j$ and then randomly interchanging modalities between those points. To see why this is a convex combination, let $I_k$ be an $(k\times k)$ identity matrix, $\mathcal{A}$ be a $(d\times d)$ block-diagonal matrix and ${\lambda }_i,\dots {\lambda }_m\in \{0,1\}$ be drawn independently from a Bernoulli distribution with $p=0.5$; then

$$\hat{\mathbf{x}}=\mathcal{A}{\mathbf{x}}_i+(I_d-\mathcal{A}){\mathbf{x}}_j\mathrm{where}\mathcal{A}=\left(\begin{array}{cccc}{\lambda }_1{I}_{d_1}& 0& \cdots & 0\\ 0& {\lambda }_2{I}_{d_2}& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\lambda }_m{I}_{d_m}\end{array}\right)$$

(16)

Each ${\lambda }_k$ chooses whether the entire modality $x_k$ is taken from ${\mathbf{x}}_i$ or from ${\mathbf{x}}_j$. To get a notion of the geometry of Modmix let ${\mathcal{X}}_i$ be the set of all modality instances of the i-th modality in D; then the Cartesian product $\mathcal{G}={\mathcal{X}}_1\times {\mathcal{X}}_2\times \cdots \times {\mathcal{X}}_m$ is all possible modality combinations. The Modmix points $\mathcal{W}\subset \mathcal{G}$ are a special subset of $\mathcal{G}$ where each point contains only modalities from at most 2 different points. To see how many points Modmix can generate, let n be the number of available points and i be the number modalities being exchanged in generating $\hat{x}$; then the number of possible new points is

$$2\sum _{i=1}^{m-1}\left(\genfrac{}{}{0pt}{}{n}{2}\right)\left(\genfrac{}{}{0pt}{}{m}{i}\right)=\left(\genfrac{}{}{0pt}{}{n}{2}\right)(2^{m+1}-2^2)$$

(17)

Consequently the growth rate $\mathsf{\Theta }\left(n^2{2}^m\right)$ of Modmix increases quadratically with the number n of original points and exponentially with the number m of modalities. In our study we are only concerned with binary classification with label $y\in \{0,1\}$, and if the two points ${\mathbf{x}}_i,{\mathbf{x}}_j$ are from different classes, then we label the generated point as follows:

$$\hat{y}=\frac{t}{m}$$

(18)

where t is the number of modalities belonging to class 1.

This type of labeling can be interpreted as majority voting without decision at the modality level. Since individual modalities are represented by separate feature vectors, each modality can be assigned its own label. When multiple modalities are combined, the resulting sample is labeled using the corresponding pain ratio. This leads to soft targets similar in form to label interpolation in mixup and mixup based works [18,19,20].

This labeling scheme penalizes too sharp decision boundaries by inducing target distributions in regions corresponding to mixed modalities with the goal of promoting stability under unseen modality deviations. As with other regularization techniques, however, this comes with no guarantee of performance improvement and may in some cases push the model away from a better solution, as has been analyzed for mixup [21,22,23].

4. X-ITE and BioVid Multimodal Datasets for Pain Detection

In this work, we use the X-ITE pain database, a multimodal dataset widely employed in pain research and automated pain recognition, particularly using facial expressions and physiological signals [24]. The dataset contains recordings from 134 subjects and includes video data as well as several physiological measurements, such as electrodermal activity (EDA), electrocardiogram (ECG), and surface electromyography (EMG).

Pain stimuli in X-ITE were applied at different intensities, targeting both tonic and phasic receptors using thermal and electrical stimulation protocols. In this study, we focus exclusively on the physiological modalities ECG, EDA, and EMG, which provide information about cardiac activity, skin conductance, and muscle activation, respectively. The EMG signals capture muscle activity from the corrugator supercilii (COR), musculus trapezius (TRA), and zygomaticus major (ZYG). These modalities result in high-dimensional feature representations, COR (82 features), ZYG (82 features), TRA (82 features), EDA (79 features), and ECG (87 features), which are concatenated to form a multimodal feature vector for each measurement. A detailed description of the feature extraction process can be found in [24].

In addition to X-ITE, we also employ the BioVid Heat Pain Database (Part A), a benchmark dataset for physiological pain recognition [25]. BioVid contains recordings from 87 healthy subjects. Pain was induced using subject-specific heat stimulation levels, ranging from baseline (no pain) to very severe pain. Consistent with our experimental design, we use only physiological signals from BioVid and exclude behavioral modalities such as facial video. The dataset includes multiple modalities such as EDA (13 features), ECG (13 features), and EMG (13 features). For our experiments the 13 features were extracted by using simple signal statistics of the raw signal.

Since this work focuses exclusively on binary pain classification, we categorize data from both datasets into two classes: no pain and pain. For X-ITE, this corresponds to the lowest and highest available pain levels, while for BioVid we use the baseline condition (BLN) and the highest pain level (PA4). All intermediate pain levels are excluded, aiming for a clear separation between classes and focusing the analysis on the most distinct physiological responses to pain.

5. Experimental Setup

5.1. Sampling Training Data

After filtering the X-ITE dataset for the lowest and highest pain levels, our database comprises $n=134$ subjects with approximately 60 feature vectors per subject. However, to reduce the computation time and create a data scarcity situation, we first randomly select only 30 subjects and then perform 10 trials for each subject in a scheme similar to cross-validation. For every trial of a given subject, we randomly take 2500 samples without replacement of the data from the remaining subjects $n-1$ (=133) as training data. In this way, we simulate a scenario in which the data may be insufficient, resulting in high model variance. One of our goals is to investigate how Modmix data augmentation can improve model performance in a way suggested by the principle of vicinal risk minimization.

A similar approach is used for BioVid, where the dataset comprises 80 subjects after excluding 7 subjects that do not contain the full set of 40 feature vectors. We evaluate the model on all 80 test subjects and randomly sample 3000 training samples without replacement from the remaining subjects.

5.2. Data Augmentation

We use Modmix as described in Section 3.2.3 to generate new data points for each subject, based on the training examples. Since Modmix can only create a finite number of new samples, the question arises whether this number is large enough. With a training sample size of only 2500 and five modalities, the number of new samples is given by

$$\left(\genfrac{}{}{0pt}{}{2500}{2}\right)(2^6-4)\approx 1.8·{10}^8$$

(19)

This shows that Modmix can create a large number of new samples in the order of tens of millions. In practice, only a small fraction of this is needed to achieve sufficient diversity for training. In this study, we will only exchange one modality at a time, leaving us still with

$$2\left(\genfrac{}{}{0pt}{}{2500}{2}\right)\left(\genfrac{}{}{0pt}{}{5}{1}\right)\approx 3·{10}^7$$

(20)

possible new samples. In the case of X-ITE even altering only one modality already changes ∼20% of the feature vector. For this reason we filter the generated data with a dedicated neural network trained on the original data and keep only the samples that meet a sufficient confidence threshold.

5.3. Neural Network Model

We configured the RBF patch network used in this study with 160 patches of dimension 16. This patch configuration was determined by a superficial grid search. The entire network, including the RBF centers and width parameters, is trained together using the stochastic gradient descent (sgd) method with Nestorov momentum. To reduce the learning time and the number of learnable parameters, we use the upper triangular matrix of the Cholesky decomposition of the covariance matrix of the Mahalanobis distance. We then combine 10 independently trained RBF patch networks with different random patch assignments into a final decision by averaging the outputs of the individual models.

6. Results

6.1. X-ITE

The main focus of this work is on experiments with X-ITE to investigate the proposed RBF patch network together with the Modmix data augmentation. We first performed all experiments with three different data variants: no augmentation, mixup, and Modmix. For a better understanding of the performance of the RBF patch network, all experiments were also conducted using the two related models, RBF-SVM and random forest. In particular, the random forest can be considered a benchmark, as it is one of the most successful, robust, and hard-to-beat ensemble models. In this work, a random forest of depth 2 was used. For each of the 30 test subjects, 10 trials were performed in a leave-one-out scheme, in which the training data in each test were randomly selected from the data excluding the test subject. In addition, both the Modmix and mixup training data contain 500 additional augmented data points each.

Figure 2 shows the average test accuracies of all models as box plots, sorted by training data variant. All models show comparable performance with median values of 85–87%. All model training data combinations show a similar distribution of accuracies, with relatively large interquartile ranges (IRQs) being noticeable. Accordingly,

Table 1. Model accuracies (%) (mean ± std) for each dataset.

Dataset	RBF-Patch-Net	RBF-SVM	Random Forest
No Aug	86.4 ± 10.3	83.7 ± 12.4	86.4 ± 10.7
Modmix	87.1 ± 11.3	85.0 ± 11.6	84.6 ± 12.3
Mixup	86.3 ± 10.7	84.8 ± 11.8	85.2 ± 11.7

shows consistently high relative standard deviations of 10–12%. This is probably due to the fact that test subjects respond differently to different stimuli and are therefore easier or more difficult to classify. This diversity is typical for medical applications.

At 87.1%, the RBF patch network in combination with Modmix shows the best performance of all combinations. However, an increase in the standard deviation of about 1% compared to no augmentation can also be observed. This suggests that Modmix may not lead to improvement in all subjects and may even lead to deterioration in some, which is confirmed by the accuracy-difference plots in Figure 3 for Modmix and mixup. The effect appears to be nearly symmetrical, with the number of improvements roughly matching the number of deteriorations, while some subjects were essentially unaffected.

Table 2. Performance comparison of the RBF network against other models under different data augmentation strategies. $\mathsf{\Delta }$ accuracy is computed as (RBF network mean accuracy) − (mean accuracy of the compared model). The p-value was determined using a Wilcoxon signed-rank test, and the confidence interval (CI) was calculated based on a t-distribution.

Dataset	Model	$\Delta$ Acc. (%)	p-Value	95% CI
No Augmentation	RBF-SVM	+2.7	$p<{10}^{-6}$	[1.8, 3.6]
	Random Forest	$-0.03$	0.479	[$-0.6$, 0.5]
Modmix	RBF-SVM	+2.3	$p<{10}^{-6}$	[1.6, 3.0]
	Random Forest	+2.4	$p<{10}^{-6}$	[1.6, 3.2]
Mixup	RBF-SVM	+1.3	0.044	[0.6, 2.0]
	Random Forest	+1.1	0.033	[0.3, 1.8]

shows a more detailed comparison of the performance of the RBF patch network with the other models using different training data variants. The RBF patch network consistently delivers a statistically significant improvement in accuracy of at least 1% over the other models for all training data variants. Only in comparison with the random forest without augmentation is no improvement observed, although the result is not statistically significant.

Table 3. Performance comparison of data augmentation strategies using Modmix as the baseline. $\mathsf{\Delta }$ accuracy is computed as (Modmix mean accuracy) − (mean accuracy of the compared dataset). The p-value was determined using a Wilcoxon signed-rank test, and the confidence interval (CI) was calculated based on a t-distribution.

Model	Dataset	$\Delta$ Acc. (%)	p-Value	95% CI
RBF-Patch-Net	No Augmentation	+0.7	0.004	[0.2, 1.1]
	Mixup	+0.8	0.007	[0.4, 1.2]
RBF-SVM	No Augmentation	+1.1	0.036	[0.2, 2.0]
	Mixup	$-0.2$	$p<{10}^{-5}$	[$-0.4$, $-0.1$]
Random Forest	No Augmentation	$-1.8$	$p<{10}^{-4}$	[$-2.5$, $-1.1$]
	Mixup	$-0.6$	0.042	[$-1.0$, $-0.1$]

, on the other hand, shows how performance changes when using Modmix in contrast to the other training data variants when using all models. For the RBF patch network, Modmix consistently delivers statistically significantly better results compared to no augmentation (+0.7%) and mixup (+0.8%). This result once again underscores that the RBF patch network improves with Modmix.

6.2. BioVid

To complement the results obtained with X-ITE and to better understand the behavior of the RBF patch network, we performed additional experiments with BioVid without data augmentation. Initial experiments suggested that data augmentation did not lead to consistent performance improvements on BioVid. We therefore focused our analysis on the non-augmented setting to gain unbiased insights into the models’ behavior.

Five runs were performed for each of the 80 test subjects. The graph in Figure 4 shows the average accuracy of both models, sorted in ascending order according to the random forest performance of a test subject. The models follow the same trend; i.e., test subjects who perform well or poorly in random forest also perform well or poorly in the RBF patch network. Interestingly, however, it can be seen that the RBF patch network significantly improves the results for some subjects (increase of approx. 10%), while for others it worsens the results to approximately the same extent. On average, these fluctuations cancel each other out, and both models have the same mean value of about 81% across all trials. These fluctuations are probably due to the significant differences between the test subjects, which were also observed in the experiments with X-ITE. The significant improvements for a subgroup of test subjects are very promising, and it is worth conducting further research to find out exactly what led to the improvements and how the poorer results can also be improved accordingly.

7. Conclusions

In this article, we introduced the novel RBF patch network and combined it with Modmix, an effective yet simple data augmentation strategy. Both the model architecture and the data strategy rely on randomization with the goal of improving performance while reducing model variance. On X-ITE, this approach proved effective, with the proposed method exceeding the accuracy of the extremely strong random forest baseline by almost 1%. At the same time, the results clearly showed that Modmix does not have a uniformly positive effect across all subjects. Given the vast number of possible combinations enabled by Modmix, this behavior leaves room for further investigation of this kind of augmentation strategy.

Additional experiments on the BioVid dataset provided further insights into the behavior of the RBF patch network. In this setting, data augmentation did not yield consistent improvements and was therefore omitted to achieve an analysis. While the RBF patch network and random forest achieved the same mean accuracy across all subjects, there was high variability between them. For a subset of subjects, the RBF patch network led to substantial performance gains, while for others, performance decreased by a similar extent. These effects cancel out on average, but they highlight the variability among subjects that was already observed on X-ITE.

From a modeling perspective, the results confirm that random feature patches can be effectively combined with RBF neurons to capture local similarities across modalities. The subject-dependent performance differences suggest that the proposed architecture highly interacts with individual signal characteristics, which motivates further study. Future work could focus on identifying the factors that lead to performance gains for some subjects and on adapting the model to all subjects accordingly. Furthermore, the way the patches are sampled and combined offers several unexplored design choices. The potential of stacking multiple patch layers remains an open and promising direction for future research.