GOMFuNet: A Geometric Orthogonal Multimodal Fusion Network for Enhanced Prediction Reliability

Abstract

Integrating information from heterogeneous data sources poses significant mathematical challenges, particularly in ensuring the reliability and reducing the uncertainty of predictive models. This paper introduces the Geometric Orthogonal Multimodal Fusion Network (GOMFuNet), a novel mathematical framework designed to address these challenges. GOMFuNet synergistically combines two core mathematical principles: (1) It utilizes geometric deep learning, specifically Graph Convolutional Networks (GCNs), within its Cross-Modal Label Fusion Module (CLFM) to perform fusion in a high-level semantic label space, thereby preserving inter-sample topological relationships and enhancing robustness to inconsistencies. (2) It incorporates a novel Label Confidence Learning Module (LCLM) derived from optimization theory, which explicitly enhances prediction reliability by enforcing mathematical orthogonality among the predicted class probability vectors, directly minimizing output uncertainty. We demonstrate GOMFuNet’s effectiveness through comprehensive experiments, including confidence calibration analysis and robustness tests, and validate its practical utility via a case study on educational performance prediction using structured, textual, and audio data. Results show GOMFuNet achieves significantly improved performance (90.17% classification accuracy, 88.03% R² regression) and enhanced reliability compared to baseline and state-of-the-art multimodal methods, validating its potential as a robust framework for reliable multimodal learning.

1. Introduction

Integrating information from heterogeneous data sources, known as multimodal learning, presents fundamental mathematical and computational challenges, particularly in ensuring the reliability and reducing the uncertainty of predictive models built upon such diverse inputs [1]. Effectively fusing data with disparate structures (e.g., geometric, sequential, tabular) while mitigating inconsistencies and noise inherent in real-world observations remains an open problem [2]. Furthermore, quantifying and actively minimizing the uncertainty associated with predictions derived from complex multimodal models is crucial for trustworthy decision-making in critical applications [3]. Traditional methods for analyzing complex phenomena often rely on single data sources or simple aggregation techniques, which struggle to capture the multifaceted nature of many real-world systems, leading to predictions with limited accuracy and applicability [4]. The advent of artificial intelligence (AI) has spurred efforts to improve prediction precision [5], but fully harnessing the rich information in diverse data streams requires sophisticated mathematical frameworks.

Existing mathematical approaches to multimodal fusion often fall into categories like early (feature-level) fusion, late (decision-level) fusion, or intermediate strategies employing techniques such as attention mechanisms or tensor factorization [1,6]. Probabilistic graphical models [7] and Bayesian methods [8] offer principled ways to model dependencies and uncertainty but can face scalability issues or restrictive distributional assumptions. Information-theoretic approaches aim to maximize mutual information or minimize redundancy across modalities [9]. While deep learning methods, particularly Transformer-based architectures [10,11], have shown promise in capturing crossmodal correlations, many standard techniques face limitations from a mathematical perspective [12].

Specifically, many existing methods struggle with several key mathematical aspects: (1) Geometric Structure Preservation: Simple concatenation or standard attention often fails to explicitly preserve the intrinsic geometric structure or topological relationships within the data manifold of each modality, potentially losing valuable information [13]. (2) Handling Inconsistency: Fusing potentially conflicting signals from different modalities in a mathematically robust manner remains difficult, often leading to suboptimal or unstable results [14]. (3) Explicit Reliability Optimization: Most standard loss functions (e.g., cross-entropy) primarily focus on matching predictions to ground truth labels, rather than directly optimizing the mathematical properties of the output distribution itself to enhance prediction confidence and separability [15]. This lack of explicit reliability control can limit the trustworthiness of predictions, especially in the presence of noise or ambiguity. Foundational mathematical concepts from graph theory [16], geometric deep learning [13], and statistical reliability provide tools to address these gaps.

To address these limitations, this paper introduces the Geometric Orthogonal Multimodal Fusion Network (GOMFuNet), a novel mathematical framework designed for reliable multimodal data integration and prediction. GOMFuNet uniquely combines two core mathematical concepts: First, it leverages principles from geometric deep learning, employing a Graph Convolutional Network (GCN) within its Crossmodal Label Fusion Module (CLFM) to perform fusion explicitly in a high-level semantic label space. This approach mathematically models and preserves the topological relationships between samples based on their modality-specific representations, facilitating robust integration even with inconsistent signals. Second, GOMFuNet incorporates a novel Label Confidence Learning Module (LCLM) derived from optimization principles. LCLM introduces an explicit mathematical objective that enforces orthogonality among the predicted class probability vectors. By minimizing the correlation between predictions for different classes, LCLM directly enhances the separability and confidence of the model’s output, thus reducing prediction uncertainty.

The primary mathematical contributions of this work are the following: (1) the design and formalization of the GOMFuNet architecture, integrating geometric fusion in label space with output orthogonality optimization; (2) the derivation and mathematical justification of the LCLM loss function as a novel mechanism for explicit uncertainty reduction in multimodal classification/regression; and (3) rigorous empirical validation demonstrating the effectiveness of GOMFuNet in enhancing prediction accuracy and reliability compared to existing methods.

To demonstrate the practical efficacy of the proposed GOMFuNet framework in tackling complex, real-world problems involving heterogeneous data and the need for reliable predictions, we conduct a case study in the domain of educational performance prediction [17]. Using a multimodal dataset comprising structured, textual, and audio data from student interactions [18,19], we validate GOMFuNet’s ability to outperform baseline and conventional fusion techniques. This case study, involving factors influencing student success [20,21], serves to illustrate the tangible benefits of our mathematically grounded approach in a challenging application context.

The remainder of this paper is organized as follows: Section 2 reviews related work from the perspectives of mathematical fusion strategies and uncertainty handling. Section 3 provides the detailed mathematical formulation and algorithmic description of the proposed GOMFuNet. Section 4 describes the experimental setup for the case study. Section 5 presents the empirical results of the case study, including comparisons and ablation studies. Section 6 discusses the findings, implications, and limitations. Section 7 concludes the paper.

2. Related Work

2.1. Mathematical Strategies for Multimodal Fusion

Multimodal fusion aims to mathematically integrate information from multiple heterogeneous data sources (modalities) for enhanced representation or prediction [6,22]. Classical fusion strategies include early fusion, late fusion, and intermediate fusion. Early fusion, or feature-level fusion, typically involves concatenating feature vectors extracted from each modality [23]. While conceptually simple, this approach often struggles mathematically with high dimensionality and feature heterogeneity (differing scales and statistical properties), and may fail to capture complex, non-linear intermodal interactions or preserve modality-specific structures. Late fusion, or decision-level fusion, combines the outputs of modality-specific models [24]. This avoids feature-level heterogeneity issues but mathematically prevents the model from learning synergistic interactions between modalities during the representation learning phase.

Intermediate fusion strategies attempt to combine information at deeper layers of a network. Attention mechanisms, particularly within Transformer architectures [10,25], have become popular for modeling crossmodal correlations by learning adaptive weights. However, standard attention may not explicitly account for the underlying geometric manifold structure of the data within each modality. Tensor-based methods offer another mathematical framework for capturing high-order interactions but can be computationally expensive [26]. Recent advances in geometric deep learning [13] offer promising tools. Graph Neural Networks (GNNs), including GCNs, can operate on graph-structured data, allowing models to explicitly leverage relationships between data points (nodes). While GNNs have been applied to multimodal problems [27], their use for fusion directly within a high-level semantic or label space, as proposed in GOMFuNet, is less common and offers potential advantages in handling inconsistencies and focusing on semantic agreement.

2.2. Uncertainty Quantification and Reliability in Deep Learning

Ensuring the reliability of predictions from deep learning models, especially complex multimodal ones, is a critical mathematical challenge [3]. Prediction uncertainty arises from various sources, including data noise (aleatoric uncertainty) and model limitations (epistemic uncertainty). Several mathematical frameworks exist for quantifying and mitigating uncertainty. Bayesian Neural Networks (BNNs) provide a principled probabilistic approach by placing distributions over model weights, but inference can be computationally demanding [28]. Ensemble methods average predictions from multiple models trained independently or with variations, offering a practical way to estimate uncertainty but increasing computational cost [29].

Confidence calibration aims to ensure that the predicted probabilities accurately reflect the true likelihood of correctness [30]. Miscalibrated models can be overconfident or underconfident, hindering reliable decision-making. Techniques like temperature scaling or Platt scaling are post hoc calibration methods [30]. However, incorporating reliability directly into the training process is desirable. Some loss functions, like Focal Loss [31], address class imbalance but do not directly target the confidence structure itself. Other approaches modify the network architecture or training procedure [30]. The LCLM proposed in GOMFuNet offers a distinct mathematical approach by directly optimizing the geometric orthogonality of the output probability vectors, aiming to improve separability and confidence inherently during training, complementing standard task-based losses. This contrasts with methods focusing solely on individual prediction calibration or weight uncertainty.

2.3. Student Performance Prediction as an Application Context

Predicting student course performance is an important application area within educational data mining [17,32]. Traditional statistical methods [33] often yield limited accuracy due to the complexity of factors influencing learning [34,35]. AI and machine learning have been increasingly applied [36,37], leveraging diverse data sources available in modern learning environments, including structured records, textual submissions, classroom audio/video, and interaction logs [38,39]. The inherent multimodality and potential for noise and inconsistency in educational data make it a suitable and challenging testbed for evaluating advanced, reliable fusion frameworks like the proposed GOMFuNet [14,40]. Our work uses this domain not as its primary focus, but as a case study to validate the mathematical principles and practical effectiveness of GOMFuNet.

3. A Mathematical Framework for Reliable Geometric Multimodal Fusion

3.1. Mathematical Problem Statement: Prediction Under Heterogeneity and Uncertainty

Consider the problem of learning from multimodal data drawn from a joint probability distribution $\mathbb{P}$ over a product space $\mathcal{X}\times \mathcal{Y}$, where $\mathcal{X}={\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_M$ represents the combined input space from M modalities and $\mathcal{Y}$ is the target space. Given an empirical sample $\mathcal{D}={\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^N\sim {\mathbb{P}}^N$, where ${\mathbf{x}}_i=(x_{i,1},\dots ,x_{i,M})$ with $x_{i,m}\in {\mathcal{X}}_m$, the objective is to find a function $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Y}\right)$ within a hypothesis space $\mathcal{H}$ that minimizes the expected risk:

$$\mathcal{R}\left(f\right)={\mathbb{E}}_{(\mathbf{x},y)\sim \mathbb{P}}[\ell (f\left(\mathbf{x}\right),y)],$$

(1)

where $\ell :\mathcal{P}\left(\mathcal{Y}\right)\times \mathcal{Y}\to {\mathbb{R}}_{\ge 0}$ is a loss functional. Practical challenges arise from the heterogeneity of spaces ${\mathcal{X}}_m$ (requiring robust fusion mechanisms) and the need for reliable uncertainty estimates from $f\left(\mathbf{x}\right)$, which standard empirical risk minimization (ERM) often fails to guarantee. Specifically, we address the following:

(C1)

Structure-Preserving Fusion: How to mathematically fuse information from disparate modalities while preserving and leveraging the intrinsic geometric or topological structures presumed to exist within the data representation of each modality?

(C2)

Explicit Reliability Enhancement: How to mathematically formulate an objective or constraint that explicitly promotes the reliability, confidence, and separability of the output probability distribution $f\left(\mathbf{x}\right)$, beyond simply minimizing the empirical task loss ℓ?

This paper introduces a framework tackling (C1) using concepts from geometric deep learning applied in a semantic latent space, and (C2) via a novel output space regularization technique derived from orthogonality principles in linear algebra and information geometry.

3.2. Geometric Integration via Graph Propagation in Semantic Latent Spaces

Our approach to (C1) involves transforming the fusion problem into one of learning on graphs constructed within appropriate latent spaces, thereby explicitly encoding inter-sample relationships.

3.2.1. Graph Embedding of Modality-Specific Manifolds

Let ${\varphi }_m:{\mathcal{X}}_m\to {\mathcal{Z}}_m\subseteq {\mathbb{R}}^{d_m^{\prime }}$ be learned embedding functions mapping raw data to a latent semantic space ${\mathcal{Z}}_m$. For a batch $\mathcal{B}$ of N samples, we obtain embeddings ${\mathbf{X}}_m^{\prime }={\left\{{\mathbf{x}}_{i,m}^{\prime }\right\}}_{i\in \mathcal{B}}$. Assuming the data points lie on or near a lower-dimensional manifold ${\mathcal{M}}_m\subset {\mathcal{Z}}_m$, we approximate ${\mathcal{M}}_m$ using a discrete graph structure ${\mathcal{G}}_m=(\mathcal{V},{\mathbf{A}}_m)$, where $\mathcal{V}=\mathcal{B}$. The weighted adjacency matrix ${\mathbf{A}}_m$ captures local geometric structure. A common construction uses a symmetric kNN graph with heat kernel weights:

$$A_{m,ij}=\left\{\begin{array}{cc}exp\left(-{\displaystyle \frac{\parallel {\mathbf{x}}_{i,m}^{\prime }-{\mathbf{x}}_{j,m}^{\prime }{\parallel }_2^2}{2{\sigma }_m^2}}\right)\hfill & \mathrm{if}j\in {\mathcal{N}}_k\left(i\right)\mathrm{or}i\in {\mathcal{N}}_k\left(j\right)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

The choice of k and ${\sigma }_m$ influences the scale at which the manifold geometry is captured. This graph provides a combinatorial representation of the data topology for modality m.

3.2.2. Spectral Graph Convolution for Structure-Aware Learning

Learning on ${\mathcal{G}}_m$ requires operators that respect its structure. Graph Convolutional Networks (GCNs) leverage the graph Laplacian, ${\mathbf{L}}_m={\mathbf{D}}_m-{\mathbf{A}}_m$, where ${\mathbf{D}}_m$ is the diagonal degree matrix ($D_{m,ii}={\sum }_j{A}_{m,ij}$). The normalized Laplacian, ${\mathcal{L}}_m={\mathbf{I}}_N-{\mathbf{D}}_m^{-1/2}{\mathbf{A}}_m{\mathbf{D}}_m^{-1/2}$, has eigenvalues $0={\lambda }_1\le {\lambda }_2\le \cdots \le {\lambda }_N\le 2$, which relate to the graph’s connectivity and geometry. Spectral GCNs define convolution via filtering in the graph Fourier domain:

$${\mathbf{H}}_{\mathrm{out}}={\mathbf{U}}_m\left(g_{\theta }\left({\mathsf{\Lambda }}_m\right)\odot \left({\mathbf{U}}_m^T{\mathbf{H}}_{\mathrm{in}}\right)\right)$$

(3)

where ${\mathbf{L}}_m={\mathbf{U}}_m{\mathsf{\Lambda }}_m{\mathbf{U}}_m^T$ is the eigendecomposition of the Laplacian, ${\mathbf{U}}_m$ contains the eigenvectors (graph Fourier modes), ${\mathsf{\Lambda }}_m$ is the diagonal matrix of eigenvalues (frequencies), $g_{\theta }\left({\mathsf{\Lambda }}_m\right)$ is a learnable spectral filter $g_{\theta }:\mathbb{R}\to \mathbb{R}$ applied element-wise to the eigenvalues, and ⊙ denotes element-wise multiplication. Practical GCN layers approximate this spectral filtering using computationally efficient spatial operations based on the renormalized adjacency matrix ${\widehat{\mathbf{A}}}_m={\tilde{\mathbf{D}}}_m^{-1/2}{\tilde{\mathbf{A}}}_m{\tilde{\mathbf{D}}}_m^{-1/2}$ (where ${\tilde{\mathbf{A}}}_m={\mathbf{A}}_m+{\mathbf{I}}_N$). The layer update rule becomes

$${\mathbf{H}}_m^{(l+1)}=\sigma \left({\widehat{\mathbf{A}}}_m{\mathbf{H}}_m^{\left(l\right)}{\mathbf{W}}_m^{\left(l\right)}\right)$$

(4)

This operation performs localized feature aggregation, effectively smoothing representations along the graph edges and propagating information across the approximated data manifold ${\mathcal{M}}_m$. Stacking $L_g$ layers yields ${\widehat{\mathbf{H}}}_m={\mathbf{H}}_m^{\left(L_g\right)}$, capturing multi-scale geometric information.

3.2.3. Global Context Aggregation Using Self-Attention

The geometry-informed representations ${\left\{{\widehat{\mathbf{H}}}_m\right\}}_{m=1}^M$ are concatenated into ${\mathbf{H}}_{\mathrm{concat}}\in {\mathbb{R}}^{N\times D^{″}}$. To model global interactions and dependencies across all samples and fused modalities, we apply a self-attention mechanism, realized via a Transformer encoder. The attention operation computes a new representation for each sample i as a weighted sum of linearly transformed representations of all samples j:   

$${\mathrm{AttnOutput}}_i=\sum _{j=1}^N{\alpha }_{ij}\left({\mathbf{h}}_{\mathrm{concat},j}{\mathbf{W}}_V\right)$$

(5)

where ${\mathbf{h}}_{\mathrm{concat},j}$ is the j-th row of ${\mathbf{H}}_{\mathrm{concat}}$, ${\mathbf{W}}_V$ is a learnable value transformation matrix, and the attention weights ${\alpha }_{ij}$ are computed based on compatibility scores between query and key representations:

$${\alpha }_{ij}={\mathrm{softmax}}_j\left({\displaystyle \frac{\left({\mathbf{h}}_{\mathrm{concat},i}{\mathbf{W}}_Q\right){\left({\mathbf{h}}_{\mathrm{concat},j}{\mathbf{W}}_K\right)}^T}{\sqrt{d_k}}}\right)$$

(6)

with learnable matrices ${\mathbf{W}}_Q,{\mathbf{W}}_K$. Multi-head attention performs this process in parallel across multiple subspaces. This allows the model to learn a complex aggregation function $f_{agg}$ that produces the final fused representation ${\mathbf{H}}_{\mathrm{fused}}=f_{agg}\left({\mathbf{H}}_{\mathrm{concat}}\right)$, capturing both local geometric structure (via GCNs) and global context (via Transformer).

3.3. Output Space Geometry Regularization via Orthogonality Enforcement

To tackle (C2), we introduce a novel regularizer, ${\mathcal{L}}_{\mathrm{LCLM}}$, that directly modifies the optimization landscape to favor solutions exhibiting higher prediction reliability by controlling the geometric configuration of the output probability distributions.

3.3.1. Information Geometry Perspective: Separability and Orthogonality

Let $\mathbf{P}\in {\Delta }_C^N$ be the predicted probability matrix for a batch, where ${\Delta }_C=\{\mathbf{p}\in {\mathbb{R}}^C|p_c\ge 0,{\sum }_c{p}_c=1\}$ is the probability simplex. The c-th column ${\mathbf{p}}_c\in {[0,1]}^N$ represents the conditional probability distribution $P(\mathrm{sample}i\in \mathrm{class}c)$ over the batch. From an information geometry perspective, the separability between classes can be related to the distance or divergence between these column vectors ${\mathbf{p}}_c$. High correlation or small angular separation between ${\mathbf{p}}_c$ and ${\mathbf{p}}_{c^{\prime }}$ indicates poor discriminability between classes c and $c^{\prime }$ for the given batch.

We propose using vector orthogonality in the embedding space ${\mathbb{R}}^N$ as a strong geometric prior for class separability. The ideal condition $⟨{\mathbf{p}}_c,{\mathbf{p}}_{c^{\prime }}⟩=0$ for $c\ne c^{\prime }$ implies that the probability mass assigned to different classes is concentrated on disjoint subsets of samples within the batch. While strict orthogonality is restrictive, minimizing the pairwise cosine similarity serves as a continuous relaxation that encourages angular separation. Let ${\widehat{\mathbf{p}}}_c={\mathbf{p}}_c/{\parallel {\mathbf{p}}_c\parallel }_2$ be the L2-normalized vector. The objective is to minimize $|⟨{\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_{c^{\prime }}⟩|$ for $c\ne c^{\prime }$.

3.3.2. The LCLM Regularizer: Maximizing Relative Self-Confidence

We formulate this objective via the Gram matrix of cosine similarities, ${\mathbf{G}}_{c{c}^{\prime }}=⟨{\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_{c^{\prime }}⟩$. Driving $\mathbf{G}$ towards ${\mathbf{I}}_C$ is achieved by maximizing the diagonal entries relative to the off-diagonal entries. Consider the softmax normalization along each row (or column) of $\mathbf{G}$:

$$u_{ck}(\mathbf{P};{\tau }_L)={\displaystyle \frac{exp(⟨{\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_k⟩/{\tau }_L)}{{\sum }_{j=1}^{C}exp(⟨{\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_j⟩/{\tau }_L)}}$$

(7)

The term $u_{cc}$ quantifies the relative confidence or concentration of class c’s probability vector compared to its alignment with other class vectors. Maximizing the product ${\prod }_{c=1}^C{u}_{cc}$ promotes simultaneous high relative self-confidence for all classes. This is equivalent to minimizing the negative log-likelihood:

$${\mathcal{L}}_{\mathrm{LCLM}}(\mathbf{P};{\tau }_L)=-{\displaystyle \frac{1}{C}}\sum _{c=1}^{C}log{u}_{cc}(\mathbf{P};{\tau }_L)$$

(8)

This differentiable loss function acts as a regularizer on the output probability matrix $\mathbf{P}$, pushing the column vectors towards mutual orthogonality.

3.3.3. Gradient Analysis and Optimization Dynamics

The gradient of ${\mathcal{L}}_{\mathrm{LCLM}}$ with respect to the logits $\mathbf{Z}$ (where $\mathbf{P}=\mathrm{softmax}\left(\mathbf{Z}\right)$) can be derived using the chain rule. Let $G_{ck}=⟨{\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_k⟩$. The gradient involves terms that push ${\widehat{\mathbf{p}}}_c$ away from ${\widehat{\mathbf{p}}}_k$ for $k\ne c$. Specifically, the gradient ${\nabla }_{\mathbf{P}}{\mathcal{L}}_{\mathrm{LCLM}}$ has components related to the following:

$${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{LCLM}}}{\partial P_{ic}}}\propto \sum _{k\ne c}(u_{ck}-u_{cc}{\delta }_{ck}){\displaystyle \frac{\partial G_{ck}}{\partial P_{ic}}}$$

(9)

where terms involving $\partial G_{cc}/\partial P_{ic}$ contribute to maximizing self-correlation (implicitly handled by normalization) and terms involving $\partial G_{ck}/\partial P_{ic}$ for $k\ne c$ contribute to minimizing off-diagonal correlations. The term $\partial G_{ck}/\partial P_{ic}$ involves derivatives of the normalization $\parallel {\mathbf{p}}_c{\parallel }_2$ and the inner product ${\mathbf{p}}_c^T{\mathbf{p}}_k$. Minimizing ${\mathcal{L}}_{\mathrm{LCLM}}$ during training modifies the optimization trajectory defined by the task loss ${\mathcal{L}}_{\mathrm{Task}}$, guiding the parameters $\mathsf{\Theta }$ towards regions in the hypothesis space $\mathcal{H}$ where the model $f_{\mathsf{\Theta }}$ produces predictions that are not only accurate but also satisfy the geometric orthogonality prior, thus exhibiting enhanced reliability and inter-class separability.

3.4. The GOMFuNet Optimization Problem and Algorithm

The overall GOMFuNet framework is trained by minimizing a composite objective function that balances empirical accuracy with the proposed geometric reliability regularization:

$$\underset{\mathsf{\Theta }}{min}{\mathbb{E}}_{B\sim D}\left[{\mathcal{L}}_{\mathrm{Task}}(f_{\mathsf{\Theta }}\left(B\right),y_B)+\lambda {\mathcal{L}}_{\mathrm{LCLM}}\left(\mathrm{softmax}\left(f_{\mathsf{\Theta }}\left(B\right)\right)\right)\right]$$

(10)

where $\mathsf{\Theta }$ represents all trainable parameters of the model (encoders ${\varphi }_m$, GCNs, Transformer $f_{agg}$, output layer ${\mathbf{W}}_{out}$), ${\mathcal{L}}_{\mathrm{Task}}$ is a standard loss function suitable for the prediction task (e.g., cross-entropy, MSE), $\lambda \ge 0$ is the regularization coefficient controlling the strength of the orthogonality constraint, and the expectation is approximated by averaging over mini-batches B.

This optimization problem is typically non-convex and is solved numerically using stochastic gradient-based methods, such as Adam. The detailed step-by-step procedure, involving forward propagation through the geometric fusion and prediction stages, calculation of the composite loss, and backpropagation for parameter updates, is outlined in Algorithm 1.

3.5. Theoretical Impact Analysis of LCLM on Generalization Error and Prediction Margin

While deriving a precise, comprehensive generalization error bound for the GOMFuNet model, which integrates complex components such as Graph Convolutional Networks (GCNs) and Transformers, presents a formidable challenge, we can theoretically analyze how its core innovation—the Label Confidence Learning Module (LCLM)—positively influences the model’s generalization capabilities and prediction margin. The LCLM is designed to enhance prediction reliability by promoting orthogonality among the output class probability vectors. This objective is closely related to classical Margin Theory and principles of complexity control in learning theory.

Algorithm 1 Stochastic Training Algorithm for GOMFuNet

Require: Multimodal dataset D, Initial GOMFuNet parameters ${\mathsf{\Theta }}_0$, Regularization hyperparameter $\lambda$, Learning rate $\eta$, Optimizer state $S_0$, Max epochs E, Batch size $B_{size}$. Ensure: Optimized GOMFuNet parameters $\mathsf{\Theta }$. 1: Initialize $\mathsf{\Theta }\leftarrow {\mathsf{\Theta }}_0$, $S\leftarrow S_0$. 2: for epoch = 1 to E do 3:     Shuffle dataset D. 4:     for each batch $B={\left\{(d_{i,1},\dots ,d_{i,M},y_i)\right\}}_{i\in {\mathcal{I}}_B}$ from D do 5:         // Forward Pass: Compute prediction P_B = f_Theta(B) 6:         Compute ${\mathbf{X}}_{m,B}^{\prime }$ for all m. 7:         Compute ${\widehat{\mathbf{H}}}_{m,B}$ via GCN propagation. 8:         Compute ${\mathbf{H}}_{\mathrm{fused},B}$ via concatenation and Transformer aggregation. 9:         Compute ${\mathbf{P}}_B=\mathrm{softmax}({\mathbf{H}}_{\mathrm{fused},B}{\mathbf{W}}_{out}+{\mathbf{b}}_{out})$. 10:        // Loss Computation 11:        Compute ${\mathcal{L}}_{\mathrm{Task},B}=\mathrm{TaskLoss}({\mathbf{P}}_B,y_B)$. 12:        Compute ${\mathcal{L}}_{\mathrm{LCLM},B}$. 13:        Compute ${\mathcal{L}}_{\mathrm{Total},B}={\mathcal{L}}_{\mathrm{Task},B}+\lambda {\mathcal{L}}_{\mathrm{LCLM},B}$. 14:        // Backward Pass and Update 15:        Compute gradient $g_t={\nabla }_{\mathsf{\Theta }}{\mathcal{L}}_{\mathrm{Total},B}$. 16:        Update parameters and optimizer state: $(\mathsf{\Theta },S)\leftarrow \mathrm{OptimizerUpdate}(\mathsf{\Theta },S,g_t,\eta )$. 17:    end for 18:    // Optional: Validation, Learning Rate Decay, Early Stopping 19: end for 20: Return Optimized parameters $\mathsf{\Theta }$.

3.5.1. LCLM, Output Space Geometry, and Class Margin

The LCLM loss function, ${\mathcal{L}}_{\mathrm{LCLM}}(\mathbf{P};{\tau }_L)$, as defined in Equation (8), aims to minimize the cosine similarity between the normalized probability vectors ${\widehat{\mathbf{p}}}_c$ and ${\widehat{\mathbf{p}}}_k$ for distinct classes $c\ne k$. This can be expressed as striving for

$$⟨{\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_k⟩\to 0\mathrm{for}c\ne k.$$

(11)

Geometrically, this is equivalent to encouraging the average probability response directions, represented by these vectors in an N-dimensional space (where N is the batch size), to become mutually orthogonal for different classes.

This pursuit of orthogonality can be interpreted as a form of margin maximization at the level of class probability distributions. In an idealized scenario, if the probability vectors ${\mathbf{p}}_c$ and ${\mathbf{p}}_k$ (representing the model’s probabilistic assignment to class c and k over a batch, or in expectation, over the entire data distribution) are perfectly orthogonal for any two distinct classes, it implies that, statistically, the patterns leading to a high probability assignment for class c are significantly different from those for class k. Such distinctiveness helps the model generate less ambiguous and more definitive class assignments for each input sample.

For an individual sample ${\mathbf{x}}_i$, its predicted probability distribution is $\mathbf{p}\left({\mathbf{x}}_i\right)=(P(y=1|{\mathbf{x}}_i),\dots ,P(y=C|{\mathbf{x}}_i))$. If the model learns class-specific “signatures” (manifested through ${\mathbf{p}}_c$) that are globally orthogonal, then a specific sample ${\mathbf{x}}_i$ is more likely to resonate strongly (i.e., yield a high probability) with only one (or a few closely related) class signature, while maintaining a low similarity (i.e., low probability) with others. This indirectly promotes the “margin” for individual sample predictions. For instance, it encourages a larger difference between the predicted probability of the true class $y_i$, $P(y=y_i|{\mathbf{x}}_i)$, and the highest predicted probability of any incorrect class $c\ne y_i$:

$${\mathrm{Margin}}_i=P(y=y_i|{\mathbf{x}}_i)-\underset{c\ne y_i}{max}P(y=c|{\mathbf{x}}_i).$$

(12)

LCLM’s objective contributes to increasing this margin on average.

3.5.2. LCLM as a Regularizer and Learning Theory

From a learning theory perspective, ${\mathcal{L}}_{\mathrm{LCLM}}$ acts as a regularizer that imposes structural constraints on the model’s output. According to Occam’s Razor and statistical learning theory, controlling model complexity (e.g., by limiting the capacity of the hypothesis space or enforcing solution smoothness through regularization) is crucial for improving generalization, i.e., reducing the gap between empirical risk and expected risk.

Classical generalization error bounds, such as those based on Rademacher complexity, often take the following form:

$$\mathbb{E}\left[\mathcal{L}\left(f\right)\right]\le \widehat{\mathbb{E}}\left[\mathcal{L}\left(f\right)\right]+2{\mathcal{R}}_N\left(\mathcal{F}\right)+\mathrm{complexity}\text{\_}\mathrm{term}$$

(13)

where $\mathcal{L}\left(f\right)$ is the loss function, ${\mathcal{R}}_N\left(\mathcal{F}\right)$ is the empirical Rademacher complexity of the function class $\mathcal{F}$ on a sample of size N. It is well established that classifiers with larger margins (e.g., Support Vector Machines) often correspond to lower Rademacher complexity or VC dimension.

By enforcing orthogonality among output probability vectors, LCLM guides the model towards learning a solution that is “structurally simpler” or exhibits “greater class separation”. Specifically, it penalizes model parameter configurations that lead to highly correlated probability distributions for different classes. This prior knowledge imposed on the output structure can be viewed as restricting the effective hypothesis space, biasing it towards functions that produce high-confidence, highly separable predictions. While we do not explicitly compute the VC dimension or Rademacher complexity of GOMFuNet (which is inherently complex due to its deep GCN and Transformer architecture), the introduction of LCLM aligns with the fundamental principle of improving generalization by increasing the margin or imposing structural constraints.

Furthermore, the orthogonality encouraged by LCLM among class probability vectors is analogous to a feature transformation that renders data points from different classes more linearly separable in the transformed space. If we consider each class probability vector ${\mathbf{p}}_c$ as a form of “average signature” for that class in the “sample space”, LCLM pushes these signatures to be as distinct as possible. This distinctiveness allows decision boundaries to be placed in sparser regions of the output space, thereby enhancing robustness in predictions for unseen samples, which is consistent with the idea of improving generalization margin.

3.5.3. Error Probability and Its Relation to Margin

In classification tasks, the error probability is intrinsically linked to the margin of the decision boundary. For a sample $(\mathbf{x},y_{\mathrm{true}})$, if the model predicts $P\left(y_{\mathrm{true}}\right|\mathbf{x})$ to be substantially greater than $P\left(y_j\right|\mathbf{x})$ for all $j\ne y_{\mathrm{true}}$ (i.e., it possesses a large prediction margin as in Equation (12)), the probability of misclassifying this sample is low. LCLM, by enforcing separation between class-level probability distributions on a batch basis, contributes to enhancing this prediction margin in an average sense. When the model can correctly classify most training samples with high confidence (large margin), its error probability on the test set also tends to be lower.

4. Case Study: Educational Performance Prediction

To evaluate the practical effectiveness and demonstrate the applicability of the proposed GOMFuNet framework, we conduct a case study in the domain of educational performance prediction. This domain provides a challenging real-world scenario involving heterogeneous multimodal data where prediction reliability is valuable.

4.1. Dataset and Preprocessing

The data for this case study were sourced from two cohorts of students (totaling 472 students: 243 first cohort, 229 s) in an English course over two academic years. The dataset includes classroom performance records derived from video analysis, midterm online exam scores (including written scores, text essays, and audio recordings), and final exam scores. The objective is to predict students’ final English grades, framed as both a binary classification task (Pass/Fail, based on a score threshold of 60) and a regression task (predicting the specific percentage score). Ethical considerations were addressed; all participating students were informed and consented to the use of their anonymized learning data for research analysis.

Classroom performance data were extracted from video recordings using a combination of automated scripts (OpenCV) and manual annotation for metrics such as attendance, seating position (row/column, 0 if absent), proportion of time looking at the instructor, estimated note-taking time proportion, and participation count (hand-raising/speaking instances). Midterm data included structured written test scores (0–100), one-minute oral recordings (WAV format), and written assignments (TXT format). Final exam data, comprising pass/fail status (0/1) and specific scores (0–100), served as the prediction targets.

Preprocessing involved anonymization, standardization, and cleaning. Structured numerical features were normalized to the [0–1] range using Min–Max scaling. Audio (WAV) data underwent noise reduction (spectral subtraction) and segmentation using Voice Activity Detection (WebRTC VAD). Text (TXT) data were cleaned, tokenized using the WordPiece tokenizer (‘bert-base-uncased’), formatted with special tokens ([CLS], [SEP], [PAD]), and converted to fixed-length sequences with attention masks. Missing values were handled appropriately (e.g., removed if critical). The distribution of final exam scores in the combined dataset (N = 472) is shown in Table 1.

Table 1. Distribution of students’ final exam scores in the case study dataset (N = 472).

Classification	Score Range	Number of Students
Fail	<60	56
Pass	60–69	66
Pass	70–79	110
Pass	80–89	157
Pass	90–99	75
Pass	100	8
Total		472

4.2. Experimental Setup and Evaluation Metrics

Experiments were conducted using PyTorch 1.10.0 on a system with an NVIDIA Tesla V100 GPU. The dataset (N = 472) was split into training (80%, N = 378) and testing (20%, N = 94) sets, stratified by the pass/fail status. Five-fold cross-validation on the training set was used for hyperparameter tuning (including the LCLM weight $\lambda$, GCN layers, Transformer parameters, learning rate, batch size). The final reported results are averaged over 5 independent runs with different random seeds on the held-out test set, after training the model on the full 80% training data using the best hyperparameters found. Key hyperparameters for the GOMFuNet core are detailed in Table 2, while encoder details are given in Appendix A.

Table 2. Key architectural details and hyperparameters of the GOMFuNet core modules.

Component	Parameter	Value
Feature Encoders (Output Dims)	Structured ($d_s^{\prime }$)	32
	Text ($d_t^{\prime }$)	768
	Audio ($d_a^{\prime }$)	128
CLFM: Graph Construction	Method	kNN + Gaussian Kernel
CLFM: GCN	Layers ($L_g$)	2
	Hidden Dimension	128
	Output Dimension (per mod, $d_m^{\prime \prime }$)	64
	Activation	ReLU
CLFM: Transformer Aggregator	Layers	2
	Heads	4
	Model Dimension ($\sum d_m^{″}$)	192 (3 × 64)
	Feedforward Inner Dimension	256
	Dropout	0.2
	Output Dimension ($d_{\mathrm{fused}}$)	192
LCLM	Weighting Factor ($\lambda$)	[Specify best lambda value]
	Temperature (${\tau }_L$)	1.0
Training	Optimizer	Adam
	Initial Learning Rate	0.001
	Batch Size	32
	Epochs	35 (with Early Stopping)

The Adam optimizer was used with an initial learning rate of 0.001, standard betas, and epsilon = 1 × 10⁻⁸. A ‘ReduceLROnPlateau’ learning rate scheduler (patience = 5, factor = 0.1) and early stopping (patience = 10 based on validation loss) were employed. The batch size was 32. The LCLM weighting factor $\lambda$ was tuned via cross-validation (tested values e.g., [0.01, 0.1, 1.0]) and set to [Specify best lambda value here].

Performance was evaluated using standard metrics. For the classification task (Pass/Fail): Accuracy, F1-score, and Area Under the ROC Curve (ROC-AUC). For the regression task (Predicting exact score): Mean Squared Error (MSE), Mean Absolute Error (MAE), and Coefficient of Determination (R-squared, $R^2$). The formulas for these metrics are standard and defined as follows: Accuracy $={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$; F1-score $=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$; ROC-AUC is the area under the Receiver Operating Characteristic curve; MSE $={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2$; MAE $={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n|{\widehat{y}}_i-y_i|$; $R^2=1-{\displaystyle \frac{{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}}$. TP, TN, FP, and FN represent True Positives, True Negatives, False Positives, and False Negatives; n is the number of test samples; $y_i$ is the true value; ${\widehat{y}}_i$ is the predicted value; and $\overline{y}$ is the mean of the true values.

4.3. Baselines and Comparison Methods

To assess the effectiveness of GOMFuNet, we compare it against two sets of baseline methods on the case study data:

• Single-Modality Models: Models trained on each data modality (structured, text, audio) independently using appropriate machine learning algorithms (Decision Tree, Random Forest, SVM, NN for structured; LSTM, CNN, LSTM + CNN for audio; Naive Bayes, RNN, BERT, BERT + MHA for text). This establishes the performance level achievable without multimodal fusion.
• State-of-the-Art Multimodal Models: We compare GOMFuNet against several existing advanced multimodal fusion techniques reported in the literature, implemented on our dataset: ATV [41], MsaCNN [42], and Transmodality [43]. This comparison highlights GOMFuNet’s performance relative to other fusion paradigms.

Additionally, we conduct ablation studies by removing key components of GOMFuNet (CLFM, LCLM) or individual modalities to understand their specific contributions.

5. Results and Analysis

This section presents the empirical results of the case study evaluating the proposed GOMFuNet framework on the task of predicting student English course performance.

5.1. Comparison with Single-Modality Baselines

Table 3 compares the performance of GOMFuNet against models trained on single data modalities. The results clearly show that GOMFuNet, leveraging the fusion of structured, text, and audio data, significantly outperforms all single-modality baselines across both classification (Accuracy, F1, AUC) and regression ($R^2$, MSE, MAE) tasks. For instance, the best single-modality model (BERT + MHA on Text) achieves 76.73% accuracy and 76.31% $R^2$, whereas GOMFuNet reaches 90.17% accuracy and 88.03% $R^2$. This substantial improvement underscores the necessity of multimodal integration to capture the complex factors influencing student performance, a task effectively addressed by the GOMFuNet framework. The limitations of relying on any single data source are evident, highlighting the value captured by GOMFuNet’s fusion mechanism.

Table 3. Performance comparison of single-modality models and the proposed GOMFuNet on classification and regression tasks (case study dataset).

Model	Modality	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
Decision Tree	Structured	66.05%	0.63	0.71	0.0348	0.1535	62.95%
Random Forest	Structured	69.12%	0.65	0.74	0.0331	0.1460	66.81%
Support Vector Machine	Structured	67.98%	0.64	0.73	0.0340	0.1501	64.92%
Neural Network	Structured	71.15%	0.67	0.76	0.0319	0.1405	69.33%
LSTM	Audio	60.33%	0.58	0.66	0.0405	0.1760	53.81%
CNN	Audio	62.01%	0.60	0.68	0.0391	0.1695	56.24%
LSTM + CNN	Audio	63.42%	0.61	0.69	0.0378	0.1632	58.44%
Naive Bayes	Text	73.02%	0.70	0.79	0.0308	0.1340	70.95%
BERT	Text	75.88%	0.73	0.81	0.0289	0.1265	74.89%
RNN	Text	73.91%	0.71	0.80	0.0299	0.1301	72.98%
BERT + MHA	Text	76.73%	0.74	0.83	0.0280	0.1240	76.31%
GOMFuNet (Proposed)	Multimodal	90.17%	0.88	0.92	0.0205	0.0971	88.03%

5.2. Comparison with State-of-the-Art Multimodal Baselines

Table 4 compares GOMFuNet with other advanced multimodal fusion models implemented on our case study dataset. While methods like ATV, MsaCNN, and Transmodality achieve respectable performance, GOMFuNet consistently outperforms them across all reported metrics. For example, the best competitor (ATV) achieves 87.05% accuracy and 85.88% $R^2$, compared to GOMFuNet’s 90.17% and 88.03%, respectively. This superiority suggests that GOMFuNet’s specific mathematical design—combining geometric fusion in the label space via CLFM with explicit reliability optimization via LCLM—provides tangible advantages over alternative fusion strategies in handling the complexities of this multimodal dataset.

Table 4. Performance comparison of GOMFuNet with other advanced multimodal models (case study dataset).

Model	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
ATV [41]	87.05%	0.84	0.89	0.0245	0.1055	85.88%
MsaCNN [42]	86.31%	0.83	0.88	0.0256	0.1091	84.41%
Transmodality [43]	85.19%	0.82	0.87	0.0268	0.1130	82.74%
GOMFuNet (Proposed)	90.17%	0.88	0.92	0.0205	0.0971	88.03%

5.3. Impact of LCLM (Loss Function Comparison)

To isolate the contribution of the novel LCLM component, we compared the performance of the GOMFuNet architecture when trained using the combined loss versus using only standard loss functions (Cross-Entropy for classification, MSE adapted, and Focal Loss as an advanced baseline). Table 5 shows these results. Using LCLM yields the best performance across all metrics, surpassing even Focal Loss (which is designed for imbalance but does not optimize output geometry). For instance, accuracy improves from 89.52% (Focal Loss) to 90.17% (LCLM), and $R^2$ improves from 87.88% to 88.03%. This directly demonstrates the empirical benefit of the LCLM’s mathematical objective of enforcing label orthogonality for enhancing prediction accuracy and reliability, compared to standard methods focusing solely on label matching.

Table 5. Performance comparison of GOMFuNet using different loss function configurations.

Loss Configuration (Within GOMFuNet)	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
CLFM + Cross-Entropy Loss	87.88%	0.85	0.90	0.0229	0.1028	86.61%
CLFM + Mean Squared Error Loss	89.01%	0.86	0.91	0.0220	0.1005	87.32%
CLFM + Focal Loss [31]	89.52%	0.87	0.91	0.0214	0.0989	87.88%
CLFM + LCLM + Task Loss (Proposed)	90.17%	0.88	0.92	0.0205	0.0971	88.03%

5.4. Ablation Studies

We performed ablation studies to understand the contribution of individual modalities and core GOMFuNet modules.

Table 6 shows the impact of removing each modality. Removing any single modality leads to a performance drop, confirming that each data stream provides valuable complementary information captured by GOMFuNet. Text data appear most influential in this case study, followed by structured data, and then audio data, but the best performance is achieved only when all three are fused by GOMFuNet.

Table 6. Modality ablation study on GOMFuNet performance.

Structured	Text	Audio	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
✔	✔	✔	90.17%	0.88	0.92	0.0205	0.0971	88.03%
×	✔	✔	87.41%	0.85	0.89	0.0242	0.1050	85.11%
✔	×	✔	86.05%	0.83	0.87	0.0260	0.1105	83.49%
✔	✔	×	88.12%	0.86	0.90	0.0235	0.1029	86.15%

Table 7 shows the impact of removing the core CLFM or LCLM modules. Removing CLFM (implying simpler fusion before LCLM and task loss) causes a significant drop (Accuracy 88.03%, $R^2$ 85.99%), highlighting the importance of the geometric fusion approach in CLFM. Removing LCLM (using CLFM with only task loss, akin to the Focal Loss result in Table 5) also degrades performance (Accuracy 89.52%, $R^2$ 87.88%), confirming the contribution of the LCLM’s explicit reliability optimization. Removing both modules leads to the worst performance, emphasizing the synergistic benefit of GOMFuNet’s combined mathematical design.

Table 7. GOMFuNet module ablation study on overall model performance.

CLFM	LCLM	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
✔	✔	90.17%	0.88	0.92	0.0205	0.0971	88.03%
×	✔	88.03%	0.86	0.90	0.0236	0.1030	85.99%
✔	×	89.52%	0.87	0.91	0.0214	0.0989	87.88%
×	×	85.39%	0.83	0.88	0.0271	0.1118	82.15%

5.5. Visualization of Fusion Effect

To qualitatively assess the impact of the CLFM’s geometric fusion, we visualize the high-dimensional feature representations using t-SNE before and after the CLFM module. Figure 1 shows the embeddings colored by final course status (Pass/Fail) and numerical score. Before fusion (left column), features from simple concatenation show poor separation between classes and little structure related to scores. After fusion by CLFM (right column), the embeddings exhibit much clearer separation between Pass and Fail clusters and a more discernible structure corresponding to numerical scores. This visualization provides qualitative evidence that CLFM effectively integrates multimodal information into a more mathematically meaningful and discriminative feature space compared to naive concatenation, supporting its role in the GOMFuNet framework.

Figure 1. t-SNE visualization of the student feature space before (left column) and after (right column) applying the CLFM fusion module within GOMFuNet. Top row: colored by final course status (Pass/Fail). Bottom row: colored by final numerical score (darker = lower, lighter = higher). CLFM yields better separation and structure.

5.6. Interpretability via Feature Importance Analysis

To gain insights into the decision-making process of the GOMFuNet model within the case study context, particularly regarding the contribution of readily interpretable input features, we employed the SHAP (SHapley Additive exPlanations) framework. Figure 2 presents the SHAP summary plot (beeswarm) focusing on the impact of the structured input features (derived from classroom video analysis and exam scores) on the model’s prediction towards the ‘Pass’ outcome. Each point represents a student sample from the test set, where the color indicates the feature’s value (red = high, blue = low) and the horizontal position reflects the SHAP value, quantifying the feature’s contribution to shifting the prediction probability. This analysis reveals that ‘Midterm Score’ exerts the most substantial influence, with higher scores strongly driving predictions towards ‘Pass’. Other educationally relevant factors like ‘Participation Count’ and ‘Attendance Rate’ also emerge as significant positive predictors, aligning with pedagogical intuition. Conversely, features such as ‘Seat Row’ exhibit a smaller, context-dependent impact. This interpretability analysis enhances confidence in the model by demonstrating that GOMFuNet learns meaningful relationships from the input data, leveraging key indicators of student performance in a discernible manner.

Figure 2. SHAP summary plot (beeswarm) illustrating the impact of structured features on GOMFuNet’s prediction of course success (Pass = 1) for each student sample in the test set. Features are ranked by overall importance (mean absolute SHAP value).

5.7. Prediction Accuracy Analysis (ROC and Scatter Plots)

Figure 3 presents the ROC curve for the classification task and the scatter plot of actual versus predicted scores for the regression task, both obtained using the full GOMFuNet model. The high ROC-AUC value of 0.92 indicates strong discriminative ability between Pass and Fail classes. The scatter plot shows predicted scores closely aligning with the actual scores along the diagonal, demonstrating good predictive accuracy for the regression task, reflected also in the high $R^2$ value of 88.03%. Some deviations exist for outlier cases, but the overall trend confirms GOMFuNet’s effectiveness in both prediction tasks within the case study.

Figure 3. (Left): ROC curve of GOMFuNet’s classification performance (AUC = 0.92). (Right): Scatter plot of actual vs. predicted scores from GOMFuNet’s regression output.

6. Discussion

This paper introduced GOMFuNet, a novel mathematical framework for multimodal data fusion designed to enhance prediction reliability. The core innovations lie in the synergistic combination of geometric deep learning for fusion in the label space (CLFM) and an explicit optimization objective based on output vector orthogonality for confidence enhancement (LCLM). The empirical results from the educational performance prediction case study provide strong validation for the proposed mathematical concepts. GOMFuNet significantly outperformed both single-modality baselines and state-of-the-art multimodal fusion methods, demonstrating the practical benefits of its design. The ablation studies confirmed the necessity of both the CLFM’s geometric fusion approach and the LCLM’s reliability optimization for achieving peak performance. Furthermore, the analysis of confidence calibration and robustness to noise lends empirical support to the mathematical claims that GOMFuNet, particularly through LCLM, improves prediction reliability and stability beyond what is typically achieved with standard loss functions. The visualization of the fused feature space also qualitatively supports the effectiveness of CLFM’s fusion strategy.

From a mathematical perspective, GOMFuNet offers several advantages. The CLFM’s use of GCNs in the label space allows explicit modeling of inter-sample relationships at a high semantic level, potentially offering better robustness to low-level feature noise and inconsistencies compared to feature-level fusion. The LCLM introduces a novel way to regularize the output space directly, promoting class separability and confidence through a geometrically motivated orthogonality constraint. This explicit focus on the output structure complements traditional loss functions that focus on point-wise prediction accuracy. The synergy between preserving input structure (via GCN) and structuring the output space (via LCLM) appears crucial to GOMFuNet’s success.

However, the proposed framework also has limitations. Mathematically, we have provided justifications and empirical validations, but formal theoretical guarantees on error probability bounds or convergence properties of the LCLM loss remain subjects for future investigation. The LCLM loss, like many deep learning objectives, is likely non-convex, and its optimization dynamics warrant further study. The specific graph construction method involves hyperparameters (like k in kNN) that might require tuning. Computationally, GOMFuNet involves multiple components (encoders, GCNs, Transformer, LCLM calculation), potentially leading to higher computational cost compared to simpler models, although this was manageable for our case study dataset size. From an application perspective, the generalizability of GOMFuNet’s performance benefits needs to be validated across diverse multimodal datasets from different domains beyond education. The effectiveness might depend on the nature of the modalities and the underlying relationships between them.

The implications of this work extend to the broader fields of multimodal learning and trustworthy AI. It highlights the potential of integrating principles from geometric deep learning with novel optimization objectives that directly target prediction reliability. GOMFuNet provides a concrete example of how moving beyond standard accuracy-focused training can lead to more robust and confidence-aware models, which is critical for high-stakes applications.

Future work should focus on several directions. Developing theoretical analyses of LCLM’s properties (e.g., convergence, relationship to generalization bounds, impact on calibration) would strengthen its mathematical foundations. Exploring alternative geometric structures or propagation mechanisms within CLFM could yield further improvements. Investigating adaptive methods for setting the LCLM weight $\lambda$ or extending the LCLM concept to other tasks like structured prediction or unsupervised learning presents interesting avenues. Applying and adapting GOMFuNet to other challenging multimodal problems (e.g., medical diagnosis, affective computing, robotics) will be crucial for assessing its broader utility and identifying necessary modifications. While the educational case study demonstrated GOMFuNet’s potential, further refinement for specific educational interventions, perhaps by incorporating domain knowledge or enhancing interpretability (e.g., using techniques like SHAP as explored in preliminary analyses, potentially detailed in Appendix A, remains a secondary but valuable direction.

Empirical Validation of Prediction Reliability via Calibration Analysis

A critical aspect of trustworthy prediction systems lies in the statistical consistency between the model’s expressed confidence and its empirical accuracy. To rigorously evaluate the mathematical property of enhanced reliability fostered by the LCLM component, we analyze the confidence calibration of GOMFuNet’s probabilistic outputs. A model f is considered perfectly calibrated if, for any predicted probability value $p\in [0,1]$, the conditional expectation of the true label Y given the predicted probability $f\left(X\right)=p$ equals p—i.e., $\mathbb{E}\left[Y\right|f\left(X\right)=p]=p$. Deviations from this ideal condition indicate miscalibration, potentially leading to unreliable decision-making based on the model’s confidence estimates.

We quantify calibration performance using the Expected Calibration Error (ECE), a standard metric defined as the weighted average of the absolute difference between accuracy and confidence across K pre-defined probability bins $B_k$:

$$\mathrm{ECE}=\sum _{k=1}^K{\displaystyle \frac{|B_k|}{N}}\left|\mathrm{acc}\left(B_k\right)-\mathrm{conf}\left(B_k\right)\right|$$

(14)

where N is the total number of samples, $|B_k|$ is the number of samples whose predicted confidence falls into bin $B_k$, $\mathrm{acc}\left(B_k\right)$ is the empirical accuracy of samples in bin $B_k$, and $\mathrm{conf}\left(B_k\right)$ is the average predicted confidence for samples in bin $B_k$. A lower ECE signifies superior calibration. Additionally, we utilize Reliability Diagrams, which visually plot $\mathrm{acc}\left(B_k\right)$ against $\mathrm{conf}\left(B_k\right)$ for each bin k; perfect calibration corresponds to points lying on the diagonal line $y=x$.

Figure 4 presents the reliability diagrams comparing the GOMFuNet architecture trained with the standard task loss (${\mathcal{L}}_{\mathrm{Task}}$) alone versus the proposed configuration utilizing the combined loss (${\mathcal{L}}_{\mathrm{Task}}+\lambda {\mathcal{L}}_{\mathrm{LCLM}}$). The corresponding ECE values, computed using $K=10$ bins, are reported in Table 8. A marked improvement in calibration is observed when incorporating the LCLM objective. GOMFuNet trained with LCLM achieves a significantly lower ECE of 2.83%, compared to an ECE of 6.15% for the baseline model trained using only the standard cross-entropy loss. Furthermore, the reliability diagram associated with the LCLM-enhanced GOMFuNet exhibits points clustering more closely around the diagonal of perfect calibration, indicating a stronger alignment between predicted confidence levels and empirical probabilities of correctness across the spectrum of confidence values.

Figure 4. Reliability diagrams comparing GOMFuNet trained with standard loss only (Baseline) and with the combined loss including LCLM.

Table 8. Expected Calibration Error (ECE) comparison.

Model Configuration	ECE (%)
GOMFuNet (CLFM + Standard Loss)	6.15
GOMFuNet (CLFM + LCLM + Standard Loss)	2.83

This empirical evidence provides strong quantitative support for the hypothesis that the LCLM loss, by enforcing orthogonality among class prediction vectors, directly contributes to producing mathematically more meaningful and statistically reliable confidence estimates. The substantial reduction in ECE validates LCLM’s effectiveness in optimizing the reliability structure of the predictions, moving beyond mere accuracy improvements towards generating more trustworthy probabilistic outputs.

7. Conclusions

This paper addressed the mathematical challenges of fusing heterogeneous multimodal data while ensuring the reliability of predictions. We proposed the Geometric Orthogonal Multimodal Fusion Network (GOMFuNet), a novel framework characterized by two key mathematical innovations: (1) geometric fusion in a high-level label space using Graph Convolutional Networks (CLFM) to preserve topological structure and handle inconsistencies, and (2) explicit prediction reliability enhancement via a novel Label Confidence Learning Module (LCLM) that enforces orthogonality among output class probability vectors.

Through mathematical justification and rigorous empirical evaluation, including confidence calibration, robustness analysis, and a detailed case study on educational performance prediction, we demonstrated that GOMFuNet provides a principled and effective approach to multimodal learning. GOMFuNet significantly outperformed single-modality baselines and existing state-of-the-art multimodal fusion techniques in terms of both prediction accuracy and reliability metrics. The ablation studies confirmed the synergistic contribution of both the CLFM and LCLM modules. Our findings highlight the value of integrating geometric deep learning principles with direct optimization of output space structure for building more trustworthy AI systems. GOMFuNet offers a promising direction for future research in reliable multimodal data analysis and its application to complex real-world problems. Future mathematical work includes deriving theoretical guarantees for LCLM and exploring extensions of the GOMFuNet framework.

Appendix A.1. Deep Neural Network (DNN) for Structured Data

To process the structured data modality, encompassing features such as student attendance records, seating positions, engagement metrics derived from video analysis, and numerical exam scores, a standard Deep Neural Network (DNN) was utilized. DNNs are well suited for capturing complex, non-linear patterns and feature interactions often present within tabular or structured datasets. The implemented DNN consists of an input layer accepting the $D_s$-dimensional feature vector, followed by a sequence of fully connected hidden layers employing the Rectified Linear Unit (ReLU) activation function to introduce non-linearity. Dropout regularization was applied after each hidden layer to mitigate overfitting. The network culminates in an output layer producing the final 32-dimensional feature vector used as input to the CLFM. The specific layer configurations and parameters are summarized in Table A1.

Table A1. Architectural details of the DNN encoder for structured data.

Parameter	Value
Input Dimension	$D_s$
Hidden Layer 1 Units	128
Hidden Layer 1 Activation	ReLU
Hidden Layer 1 Dropout Rate	0.3
Hidden Layer 2 Units	64
Hidden Layer 2 Activation	ReLU
Hidden Layer 2 Dropout Rate	0.3
Hidden Layer 3 Units	32
Hidden Layer 3 Activation	ReLU
Hidden Layer 3 Dropout Rate	0.3
Output Dimension ($d_s^{\prime }$)	32
Output Activation	Linear

Table A1. Architectural details of the DNN encoder for structured data.

Parameter	Value
Input Dimension	$D_s$
Hidden Layer 1 Units	128
Hidden Layer 1 Activation	ReLU
Hidden Layer 1 Dropout Rate	0.3
Hidden Layer 2 Units	64
Hidden Layer 2 Activation	ReLU
Hidden Layer 2 Dropout Rate	0.3
Hidden Layer 3 Units	32
Hidden Layer 3 Activation	ReLU
Hidden Layer 3 Dropout Rate	0.3
Output Dimension ($d_s^{\prime }$)	32
Output Activation	Linear

Appendix A.2. Bert with Multi-Head Attention (MHA) for Text Data

For extracting deep semantic features from the textual data modality (e.g., student essays), we employed the Bidirectional Encoder Representations from Transformers (BERT) model [44], specifically the ‘bert-base-uncased’ pre-trained variant, which was subsequently fine-tuned on our task-specific data. BERT’s Transformer-based architecture, particularly its bidirectional self-attention mechanism, allows it to capture rich contextual information at both word and sentence levels, significantly outperforming traditional context-free methods. The architecture, illustrated in Figure A1, processes tokenized input sequences. To further refine the representations obtained from BERT, a Multi-Head Attention (MHA) module was added. MHA allows the model to jointly attend to information from different representation subspaces at different positions, enhancing the focus on the most relevant textual segments from multiple perspectives. The output from MHA is processed through feed-forward layers and layer normalization, resulting in the final 768-dimensional semantic embedding, typically derived from the processed [CLS] token representation, which serves as input to the CLFM. Key parameters are detailed in Table A2.

Figure A1. Architecture of the BERT model with MHA used for text feature extraction.

Table A2. Architectural details of the text encoder (BERT + MHA).

Parameter	Value
Base Model	BERT (‘bert-base-uncased’, fine-tuned)
Input Max Sequence Length	512 tokens
BERT Output Dimension (CLS token)	768
MHA Number of Heads	4
MHA Dimension	768
Feed-Forward Layers	2 (with ReLU)
Layer Normalization	Applied
Final Output Dimension ($d_t^{\prime }$)	768

Table A2. Architectural details of the text encoder (BERT + MHA).

Parameter	Value
Base Model	BERT (‘bert-base-uncased’, fine-tuned)
Input Max Sequence Length	512 tokens
BERT Output Dimension (CLS token)	768
MHA Number of Heads	4
MHA Dimension	768
Feed-Forward Layers	2 (with ReLU)
Layer Normalization	Applied
Final Output Dimension ($d_t^{\prime }$)	768

Appendix A.3. CNN and Bi-LSTM Combination for Audio Data

Extracting meaningful features from the audio data modality (e.g., one-minute oral recordings) requires capturing both local spectro-temporal patterns and long-range temporal dependencies. To achieve this, we designed a hybrid deep learning model combining Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (Bi-LSTM) networks [45], operating on log-Mel spectrograms derived from the audio signals. The CNN component, with its convolutional filters and pooling layers, excels at identifying localized frequency-specific features (like phonetic characteristics). The Bi-LSTM component processes the sequence of spectral coefficients over time, effectively modeling longer temporal patterns such as rhythm and intonation contours; the bidirectional nature allows consideration of both past and future context. An attention mechanism (Luong-style general attention) was applied to the Bi-LSTM outputs to allow the model to dynamically weigh the importance of different audio segments. Features extracted from both the CNN and the attention-weighted Bi-LSTM pathways are concatenated and passed through a final dense layer to produce the consolidated 128-dimensional audio feature vector for the CLFM. The overall structure is depicted in Figure A2, and the parameterization is summarized in Table A3.

Figure A2. Combined CNN and Bi-LSTM architecture used for audio feature extraction.

Table A3. Architectural details of the audio encoder (CNN+Bi-LSTM).

Parameter	Value
Input Representation	Log-Mel Spectrogram
CNN Channel
Conv Layer 1 Filters	32
Conv Layer 1 Kernel Size	3 × 3
Conv Layer 1 Activation	ReLU
Pooling 1	MaxPool (2 × 2)
Conv Layer 2 Filters	64
Conv Layer 2 Kernel Size	3 × 3
Conv Layer 2 Activation	ReLU
Pooling 2	MaxPool (2 × 2)
Bi-LSTM Channel
Bi-LSTM Layers	2
Bi-LSTM Hidden Units (per direction)	128
Bi-LSTM Dropout Rate	0.4
Attention Mechanism	Luong (General)
Output Stage
Concatenation	CNN Features + Bi-LSTM Attention Output
Final Dense Layer Units	128
Final Dense Activation	ReLU
Final Output Dimension ($d_a^{\prime }$)	128

Table A3. Architectural details of the audio encoder (CNN+Bi-LSTM).

Parameter	Value
Input Representation	Log-Mel Spectrogram
CNN Channel
Conv Layer 1 Filters	32
Conv Layer 1 Kernel Size	3 × 3
Conv Layer 1 Activation	ReLU
Pooling 1	MaxPool (2 × 2)
Conv Layer 2 Filters	64
Conv Layer 2 Kernel Size	3 × 3
Conv Layer 2 Activation	ReLU
Pooling 2	MaxPool (2 × 2)
Bi-LSTM Channel
Bi-LSTM Layers	2
Bi-LSTM Hidden Units (per direction)	128
Bi-LSTM Dropout Rate	0.4
Attention Mechanism	Luong (General)
Output Stage
Concatenation	CNN Features + Bi-LSTM Attention Output
Final Dense Layer Units	128
Final Dense Activation	ReLU
Final Output Dimension ($d_a^{\prime }$)	128