GOMFuNet: A Geometric Orthogonal Multimodal Fusion Network for Enhanced Prediction Reliability

Abstract

Integrating information from heterogeneous data sources poses significant mathematical challenges, particularly in ensuring the reliability and reducing the uncertainty of predictive models. This paper introduces the Geometric Orthogonal Multimodal Fusion Network (GOMFuNet), a novel mathematical framework designed to address these challenges. GOMFuNet synergistically combines two core mathematical principles: (1) It utilizes geometric deep learning, specifically Graph Convolutional Networks (GCNs), within its Cross-Modal Label Fusion Module (CLFM) to perform fusion in a high-level semantic label space, thereby preserving inter-sample topological relationships and enhancing robustness to inconsistencies. (2) It incorporates a novel Label Confidence Learning Module (LCLM) derived from optimization theory, which explicitly enhances prediction reliability by enforcing mathematical orthogonality among the predicted class probability vectors, directly minimizing output uncertainty. We demonstrate GOMFuNet’s effectiveness through comprehensive experiments, including confidence calibration analysis and robustness tests, and validate its practical utility via a case study on educational performance prediction using structured, textual, and audio data. Results show GOMFuNet achieves significantly improved performance (90.17% classification accuracy, 88.03% R² regression) and enhanced reliability compared to baseline and state-of-the-art multimodal methods, validating its potential as a robust framework for reliable multimodal learning.

1. Introduction

Integrating information from heterogeneous data sources, known as multimodal learning, presents fundamental mathematical and computational challenges, particularly in ensuring the reliability and reducing the uncertainty of predictive models built upon such diverse inputs [1]. Effectively fusing data with disparate structures (e.g., geometric, sequential, tabular) while mitigating inconsistencies and noise inherent in real-world observations remains an open problem [2]. Furthermore, quantifying and actively minimizing the uncertainty associated with predictions derived from complex multimodal models is crucial for trustworthy decision-making in critical applications [3]. Traditional methods for analyzing complex phenomena often rely on single data sources or simple aggregation techniques, which struggle to capture the multifaceted nature of many real-world systems, leading to predictions with limited accuracy and applicability [4]. The advent of artificial intelligence (AI) has spurred efforts to improve prediction precision [5], but fully harnessing the rich information in diverse data streams requires sophisticated mathematical frameworks.

Existing mathematical approaches to multimodal fusion often fall into categories like early (feature-level) fusion, late (decision-level) fusion, or intermediate strategies employing techniques such as attention mechanisms or tensor factorization [1,6]. Probabilistic graphical models [7] and Bayesian methods [8] offer principled ways to model dependencies and uncertainty but can face scalability issues or restrictive distributional assumptions. Information-theoretic approaches aim to maximize mutual information or minimize redundancy across modalities [9]. While deep learning methods, particularly Transformer-based architectures [10,11], have shown promise in capturing crossmodal correlations, many standard techniques face limitations from a mathematical perspective [12].

Specifically, many existing methods struggle with several key mathematical aspects: (1) Geometric Structure Preservation: Simple concatenation or standard attention often fails to explicitly preserve the intrinsic geometric structure or topological relationships within the data manifold of each modality, potentially losing valuable information [13]. (2) Handling Inconsistency: Fusing potentially conflicting signals from different modalities in a mathematically robust manner remains difficult, often leading to suboptimal or unstable results [14]. (3) Explicit Reliability Optimization: Most standard loss functions (e.g., cross-entropy) primarily focus on matching predictions to ground truth labels, rather than directly optimizing the mathematical properties of the output distribution itself to enhance prediction confidence and separability [15]. This lack of explicit reliability control can limit the trustworthiness of predictions, especially in the presence of noise or ambiguity. Foundational mathematical concepts from graph theory [16], geometric deep learning [13], and statistical reliability provide tools to address these gaps.

To address these limitations, this paper introduces the Geometric Orthogonal Multimodal Fusion Network (GOMFuNet), a novel mathematical framework designed for reliable multimodal data integration and prediction. GOMFuNet uniquely combines two core mathematical concepts: First, it leverages principles from geometric deep learning, employing a Graph Convolutional Network (GCN) within its Crossmodal Label Fusion Module (CLFM) to perform fusion explicitly in a high-level semantic label space. This approach mathematically models and preserves the topological relationships between samples based on their modality-specific representations, facilitating robust integration even with inconsistent signals. Second, GOMFuNet incorporates a novel Label Confidence Learning Module (LCLM) derived from optimization principles. LCLM introduces an explicit mathematical objective that enforces orthogonality among the predicted class probability vectors. By minimizing the correlation between predictions for different classes, LCLM directly enhances the separability and confidence of the model’s output, thus reducing prediction uncertainty.

The primary mathematical contributions of this work are the following: (1) the design and formalization of the GOMFuNet architecture, integrating geometric fusion in label space with output orthogonality optimization; (2) the derivation and mathematical justification of the LCLM loss function as a novel mechanism for explicit uncertainty reduction in multimodal classification/regression; and (3) rigorous empirical validation demonstrating the effectiveness of GOMFuNet in enhancing prediction accuracy and reliability compared to existing methods.

To demonstrate the practical efficacy of the proposed GOMFuNet framework in tackling complex, real-world problems involving heterogeneous data and the need for reliable predictions, we conduct a case study in the domain of educational performance prediction [17]. Using a multimodal dataset comprising structured, textual, and audio data from student interactions [18,19], we validate GOMFuNet’s ability to outperform baseline and conventional fusion techniques. This case study, involving factors influencing student success [20,21], serves to illustrate the tangible benefits of our mathematically grounded approach in a challenging application context.

The remainder of this paper is organized as follows: Section 2 reviews related work from the perspectives of mathematical fusion strategies and uncertainty handling. Section 3 provides the detailed mathematical formulation and algorithmic description of the proposed GOMFuNet. Section 4 describes the experimental setup for the case study. Section 5 presents the empirical results of the case study, including comparisons and ablation studies. Section 6 discusses the findings, implications, and limitations. Section 7 concludes the paper.

2. Related Work

2.1. Mathematical Strategies for Multimodal Fusion

Multimodal fusion aims to mathematically integrate information from multiple heterogeneous data sources (modalities) for enhanced representation or prediction [6,22]. Classical fusion strategies include early fusion, late fusion, and intermediate fusion. Early fusion, or feature-level fusion, typically involves concatenating feature vectors extracted from each modality [23]. While conceptually simple, this approach often struggles mathematically with high dimensionality and feature heterogeneity (differing scales and statistical properties), and may fail to capture complex, non-linear intermodal interactions or preserve modality-specific structures. Late fusion, or decision-level fusion, combines the outputs of modality-specific models [24]. This avoids feature-level heterogeneity issues but mathematically prevents the model from learning synergistic interactions between modalities during the representation learning phase.

Intermediate fusion strategies attempt to combine information at deeper layers of a network. Attention mechanisms, particularly within Transformer architectures [10,25], have become popular for modeling crossmodal correlations by learning adaptive weights. However, standard attention may not explicitly account for the underlying geometric manifold structure of the data within each modality. Tensor-based methods offer another mathematical framework for capturing high-order interactions but can be computationally expensive [26]. Recent advances in geometric deep learning [13] offer promising tools. Graph Neural Networks (GNNs), including GCNs, can operate on graph-structured data, allowing models to explicitly leverage relationships between data points (nodes). While GNNs have been applied to multimodal problems [27], their use for fusion directly within a high-level semantic or label space, as proposed in GOMFuNet, is less common and offers potential advantages in handling inconsistencies and focusing on semantic agreement.

2.2. Uncertainty Quantification and Reliability in Deep Learning

Ensuring the reliability of predictions from deep learning models, especially complex multimodal ones, is a critical mathematical challenge [3]. Prediction uncertainty arises from various sources, including data noise (aleatoric uncertainty) and model limitations (epistemic uncertainty). Several mathematical frameworks exist for quantifying and mitigating uncertainty. Bayesian Neural Networks (BNNs) provide a principled probabilistic approach by placing distributions over model weights, but inference can be computationally demanding [28]. Ensemble methods average predictions from multiple models trained independently or with variations, offering a practical way to estimate uncertainty but increasing computational cost [29].

Confidence calibration aims to ensure that the predicted probabilities accurately reflect the true likelihood of correctness [30]. Miscalibrated models can be overconfident or underconfident, hindering reliable decision-making. Techniques like temperature scaling or Platt scaling are post hoc calibration methods [30]. However, incorporating reliability directly into the training process is desirable. Some loss functions, like Focal Loss [31], address class imbalance but do not directly target the confidence structure itself. Other approaches modify the network architecture or training procedure [30]. The LCLM proposed in GOMFuNet offers a distinct mathematical approach by directly optimizing the geometric orthogonality of the output probability vectors, aiming to improve separability and confidence inherently during training, complementing standard task-based losses. This contrasts with methods focusing solely on individual prediction calibration or weight uncertainty.

2.3. Student Performance Prediction as an Application Context

Predicting student course performance is an important application area within educational data mining [17,32]. Traditional statistical methods [33] often yield limited accuracy due to the complexity of factors influencing learning [34,35]. AI and machine learning have been increasingly applied [36,37], leveraging diverse data sources available in modern learning environments, including structured records, textual submissions, classroom audio/video, and interaction logs [38,39]. The inherent multimodality and potential for noise and inconsistency in educational data make it a suitable and challenging testbed for evaluating advanced, reliable fusion frameworks like the proposed GOMFuNet [14,40]. Our work uses this domain not as its primary focus, but as a case study to validate the mathematical principles and practical effectiveness of GOMFuNet.

3. A Mathematical Framework for Reliable Geometric Multimodal Fusion

3.1. Mathematical Problem Statement: Prediction Under Heterogeneity and Uncertainty

Consider the problem of learning from multimodal data drawn from a joint probability distribution $\mathbb{P}$ over a product space $\mathcal{X}\times \mathcal{Y}$, where $\mathcal{X}={\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_M$ represents the combined input space from M modalities and $\mathcal{Y}$ is the target space. Given an empirical sample $\mathcal{D}={\left\{({\mathbf{x}}_i,y_i)\right\}}_{i=1}^N\sim {\mathbb{P}}^N$, where ${\mathbf{x}}_i=(x_{i,1},\dots ,x_{i,M})$ with $x_{i,m}\in {\mathcal{X}}_m$, the objective is to find a function $f:\mathcal{X}\to \mathcal{P}\left(\mathcal{Y}\right)$ within a hypothesis space $\mathcal{H}$ that minimizes the expected risk:

$$\mathcal{R}\left(f\right)={\mathbb{E}}_{(\mathbf{x},y)\sim \mathbb{P}}[\ell (f\left(\mathbf{x}\right),y)],$$

(1)

where $\ell :\mathcal{P}\left(\mathcal{Y}\right)\times \mathcal{Y}\to {\mathbb{R}}_{\ge 0}$ is a loss functional. Practical challenges arise from the heterogeneity of spaces ${\mathcal{X}}_m$ (requiring robust fusion mechanisms) and the need for reliable uncertainty estimates from $f\left(\mathbf{x}\right)$, which standard empirical risk minimization (ERM) often fails to guarantee. Specifically, we address the following:

(C1)

Structure-Preserving Fusion: How to mathematically fuse information from disparate modalities while preserving and leveraging the intrinsic geometric or topological structures presumed to exist within the data representation of each modality?

(C2)

Explicit Reliability Enhancement: How to mathematically formulate an objective or constraint that explicitly promotes the reliability, confidence, and separability of the output probability distribution $f\left(\mathbf{x}\right)$, beyond simply minimizing the empirical task loss ℓ?

This paper introduces a framework tackling (C1) using concepts from geometric deep learning applied in a semantic latent space, and (C2) via a novel output space regularization technique derived from orthogonality principles in linear algebra and information geometry.

3.2. Geometric Integration via Graph Propagation in Semantic Latent Spaces

Our approach to (C1) involves transforming the fusion problem into one of learning on graphs constructed within appropriate latent spaces, thereby explicitly encoding inter-sample relationships.

3.2.1. Graph Embedding of Modality-Specific Manifolds

Let ${\varphi }_m:{\mathcal{X}}_m\to {\mathcal{Z}}_m\subseteq {\mathbb{R}}^{d_m^{\prime }}$ be learned embedding functions mapping raw data to a latent semantic space ${\mathcal{Z}}_m$. For a batch $\mathcal{B}$ of N samples, we obtain embeddings ${\mathbf{X}}_m^{\prime }={\left\{{\mathbf{x}}_{i,m}^{\prime }\right\}}_{i\in \mathcal{B}}$. Assuming the data points lie on or near a lower-dimensional manifold ${\mathcal{M}}_m\subset {\mathcal{Z}}_m$, we approximate ${\mathcal{M}}_m$ using a discrete graph structure ${\mathcal{G}}_m=(\mathcal{V},{\mathbf{A}}_m)$, where $\mathcal{V}=\mathcal{B}$. The weighted adjacency matrix ${\mathbf{A}}_m$ captures local geometric structure. A common construction uses a symmetric kNN graph with heat kernel weights:

$$A_{m,ij}=\left\{\begin{array}{cc}exp\left(-{\displaystyle \frac{\parallel {\mathbf{x}}_{i,m}^{\prime }-{\mathbf{x}}_{j,m}^{\prime }{\parallel }_2^2}{2{\sigma }_m^2}}\right)\hfill & \mathrm{if}j\in {\mathcal{N}}_k\left(i\right)\mathrm{or}i\in {\mathcal{N}}_k\left(j\right)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

The choice of k and ${\sigma }_m$ influences the scale at which the manifold geometry is captured. This graph provides a combinatorial representation of the data topology for modality m.

3.2.2. Spectral Graph Convolution for Structure-Aware Learning

Learning on ${\mathcal{G}}_m$ requires operators that respect its structure. Graph Convolutional Networks (GCNs) leverage the graph Laplacian, ${\mathbf{L}}_m={\mathbf{D}}_m-{\mathbf{A}}_m$, where ${\mathbf{D}}_m$ is the diagonal degree matrix ($D_{m,ii}={\sum }_j{A}_{m,ij}$). The normalized Laplacian, ${\mathcal{L}}_m={\mathbf{I}}_N-{\mathbf{D}}_m^{-1/2}{\mathbf{A}}_m{\mathbf{D}}_m^{-1/2}$, has eigenvalues $0={\lambda }_1\le {\lambda }_2\le \cdots \le {\lambda }_N\le 2$, which relate to the graph’s connectivity and geometry. Spectral GCNs define convolution via filtering in the graph Fourier domain:

$${\mathbf{H}}_{\mathrm{out}}={\mathbf{U}}_m\left(g_{\theta }\left({\mathsf{\Lambda }}_m\right)\odot \left({\mathbf{U}}_m^T{\mathbf{H}}_{\mathrm{in}}\right)\right)$$

(3)

where ${\mathbf{L}}_m={\mathbf{U}}_m{\mathsf{\Lambda }}_m{\mathbf{U}}_m^T$ is the eigendecomposition of the Laplacian, ${\mathbf{U}}_m$ contains the eigenvectors (graph Fourier modes), ${\mathsf{\Lambda }}_m$ is the diagonal matrix of eigenvalues (frequencies), $g_{\theta }\left({\mathsf{\Lambda }}_m\right)$ is a learnable spectral filter $g_{\theta }:\mathbb{R}\to \mathbb{R}$ applied element-wise to the eigenvalues, and ⊙ denotes element-wise multiplication. Practical GCN layers approximate this spectral filtering using computationally efficient spatial operations based on the renormalized adjacency matrix ${\widehat{\mathbf{A}}}_m={\tilde{\mathbf{D}}}_m^{-1/2}{\tilde{\mathbf{A}}}_m{\tilde{\mathbf{D}}}_m^{-1/2}$ (where ${\tilde{\mathbf{A}}}_m={\mathbf{A}}_m+{\mathbf{I}}_N$). The layer update rule becomes

$${\mathbf{H}}_m^{(l+1)}=\sigma \left({\widehat{\mathbf{A}}}_m{\mathbf{H}}_m^{\left(l\right)}{\mathbf{W}}_m^{\left(l\right)}\right)$$

(4)

This operation performs localized feature aggregation, effectively smoothing representations along the graph edges and propagating information across the approximated data manifold ${\mathcal{M}}_m$. Stacking $L_g$ layers yields ${\widehat{\mathbf{H}}}_m={\mathbf{H}}_m^{\left(L_g\right)}$, capturing multi-scale geometric information.

3.2.3. Global Context Aggregation Using Self-Attention

The geometry-informed representations ${\left\{{\widehat{\mathbf{H}}}_m\right\}}_{m=1}^M$ are concatenated into ${\mathbf{H}}_{\mathrm{concat}}\in {\mathbb{R}}^{N\times D^{″}}$. To model global interactions and dependencies across all samples and fused modalities, we apply a self-attention mechanism, realized via a Transformer encoder. The attention operation computes a new representation for each sample i as a weighted sum of linearly transformed representations of all samples j:   

$${\mathrm{AttnOutput}}_i=\sum _{j=1}^N{\alpha }_{ij}\left({\mathbf{h}}_{\mathrm{concat},j}{\mathbf{W}}_V\right)$$

(5)

where ${\mathbf{h}}_{\mathrm{concat},j}$ is the j-th row of ${\mathbf{H}}_{\mathrm{concat}}$, ${\mathbf{W}}_V$ is a learnable value transformation matrix, and the attention weights ${\alpha }_{ij}$ are computed based on compatibility scores between query and key representations:

$${\alpha }_{ij}={\mathrm{softmax}}_j\left({\displaystyle \frac{\left({\mathbf{h}}_{\mathrm{concat},i}{\mathbf{W}}_Q\right){\left({\mathbf{h}}_{\mathrm{concat},j}{\mathbf{W}}_K\right)}^T}{\sqrt{d_k}}}\right)$$

(6)

with learnable matrices ${\mathbf{W}}_Q,{\mathbf{W}}_K$. Multi-head attention performs this process in parallel across multiple subspaces. This allows the model to learn a complex aggregation function $f_{agg}$ that produces the final fused representation ${\mathbf{H}}_{\mathrm{fused}}=f_{agg}\left({\mathbf{H}}_{\mathrm{concat}}\right)$, capturing both local geometric structure (via GCNs) and global context (via Transformer).

3.3. Output Space Geometry Regularization via Orthogonality Enforcement

To tackle (C2), we introduce a novel regularizer, ${\mathcal{L}}_{\mathrm{LCLM}}$, that directly modifies the optimization landscape to favor solutions exhibiting higher prediction reliability by controlling the geometric configuration of the output probability distributions.

3.3.1. Information Geometry Perspective: Separability and Orthogonality

Let $\mathbf{P}\in {\Delta }_C^N$ be the predicted probability matrix for a batch, where ${\Delta }_C=\{\mathbf{p}\in {\mathbb{R}}^C|p_c\ge 0,{\sum }_c{p}_c=1\}$ is the probability simplex. The c-th column ${\mathbf{p}}_c\in {[0,1]}^N$ represents the conditional probability distribution $P(\mathrm{sample}i\in \mathrm{class}c)$ over the batch. From an information geometry perspective, the separability between classes can be related to the distance or divergence between these column vectors ${\mathbf{p}}_c$. High correlation or small angular separation between ${\mathbf{p}}_c$ and ${\mathbf{p}}_{c^{\prime }}$ indicates poor discriminability between classes c and $c^{\prime }$ for the given batch.

We propose using vector orthogonality in the embedding space ${\mathbb{R}}^N$ as a strong geometric prior for class separability. The ideal condition $\langle {\mathbf{p}}_c,{\mathbf{p}}_{c^{\prime }}\rangle =0$ for $c\ne c^{\prime }$ implies that the probability mass assigned to different classes is concentrated on disjoint subsets of samples within the batch. While strict orthogonality is restrictive, minimizing the pairwise cosine similarity serves as a continuous relaxation that encourages angular separation. Let ${\widehat{\mathbf{p}}}_c={\mathbf{p}}_c/{\parallel {\mathbf{p}}_c\parallel }_2$ be the L2-normalized vector. The objective is to minimize $|\langle {\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_{c^{\prime }}\rangle |$ for $c\ne c^{\prime }$.

3.3.2. The LCLM Regularizer: Maximizing Relative Self-Confidence

We formulate this objective via the Gram matrix of cosine similarities, ${\mathbf{G}}_{c{c}^{\prime }}=\langle {\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_{c^{\prime }}\rangle$. Driving $\mathbf{G}$ towards ${\mathbf{I}}_C$ is achieved by maximizing the diagonal entries relative to the off-diagonal entries. Consider the softmax normalization along each row (or column) of $\mathbf{G}$:

$$u_{ck}(\mathbf{P};{\tau }_L)={\displaystyle \frac{exp(\langle {\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_k\rangle /{\tau }_L)}{{\sum }_{j=1}^{C}exp(\langle {\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_j\rangle /{\tau }_L)}}$$

(7)

The term $u_{cc}$ quantifies the relative confidence or concentration of class c’s probability vector compared to its alignment with other class vectors. Maximizing the product ${\prod }_{c=1}^C{u}_{cc}$ promotes simultaneous high relative self-confidence for all classes. This is equivalent to minimizing the negative log-likelihood:

$${\mathcal{L}}_{\mathrm{LCLM}}(\mathbf{P};{\tau }_L)=-{\displaystyle \frac{1}{C}}\sum _{c=1}^{C}log{u}_{cc}(\mathbf{P};{\tau }_L)$$

(8)

This differentiable loss function acts as a regularizer on the output probability matrix $\mathbf{P}$, pushing the column vectors towards mutual orthogonality.

3.3.3. Gradient Analysis and Optimization Dynamics

The gradient of ${\mathcal{L}}_{\mathrm{LCLM}}$ with respect to the logits $\mathbf{Z}$ (where $\mathbf{P}=\mathrm{softmax}\left(\mathbf{Z}\right)$) can be derived using the chain rule. Let $G_{ck}=\langle {\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_k\rangle$. The gradient involves terms that push ${\widehat{\mathbf{p}}}_c$ away from ${\widehat{\mathbf{p}}}_k$ for $k\ne c$. Specifically, the gradient ${\nabla }_{\mathbf{P}}{\mathcal{L}}_{\mathrm{LCLM}}$ has components related to the following:

$${\displaystyle \frac{\partial {\mathcal{L}}_{\mathrm{LCLM}}}{\partial P_{ic}}}\propto \sum _{k\ne c}(u_{ck}-u_{cc}{\delta }_{ck}){\displaystyle \frac{\partial G_{ck}}{\partial P_{ic}}}$$

(9)

where terms involving $\partial G_{cc}/\partial P_{ic}$ contribute to maximizing self-correlation (implicitly handled by normalization) and terms involving $\partial G_{ck}/\partial P_{ic}$ for $k\ne c$ contribute to minimizing off-diagonal correlations. The term $\partial G_{ck}/\partial P_{ic}$ involves derivatives of the normalization $\parallel {\mathbf{p}}_c{\parallel }_2$ and the inner product ${\mathbf{p}}_c^T{\mathbf{p}}_k$. Minimizing ${\mathcal{L}}_{\mathrm{LCLM}}$ during training modifies the optimization trajectory defined by the task loss ${\mathcal{L}}_{\mathrm{Task}}$, guiding the parameters $\mathsf{\Theta }$ towards regions in the hypothesis space $\mathcal{H}$ where the model $f_{\mathsf{\Theta }}$ produces predictions that are not only accurate but also satisfy the geometric orthogonality prior, thus exhibiting enhanced reliability and inter-class separability.

3.4. The GOMFuNet Optimization Problem and Algorithm

The overall GOMFuNet framework is trained by minimizing a composite objective function that balances empirical accuracy with the proposed geometric reliability regularization:

$$\underset{\mathsf{\Theta }}{min}{\mathbb{E}}_{B\sim D}\left[{\mathcal{L}}_{\mathrm{Task}}(f_{\mathsf{\Theta }}\left(B\right),y_B)+\lambda {\mathcal{L}}_{\mathrm{LCLM}}\left(\mathrm{softmax}\left(f_{\mathsf{\Theta }}\left(B\right)\right)\right)\right]$$

(10)

where $\mathsf{\Theta }$ represents all trainable parameters of the model (encoders ${\varphi }_m$, GCNs, Transformer $f_{agg}$, output layer ${\mathbf{W}}_{out}$), ${\mathcal{L}}_{\mathrm{Task}}$ is a standard loss function suitable for the prediction task (e.g., cross-entropy, MSE), $\lambda \ge 0$ is the regularization coefficient controlling the strength of the orthogonality constraint, and the expectation is approximated by averaging over mini-batches B.

This optimization problem is typically non-convex and is solved numerically using stochastic gradient-based methods, such as Adam. The detailed step-by-step procedure, involving forward propagation through the geometric fusion and prediction stages, calculation of the composite loss, and backpropagation for parameter updates, is outlined in Algorithm 1.

3.5. Theoretical Impact Analysis of LCLM on Generalization Error and Prediction Margin

While deriving a precise, comprehensive generalization error bound for the GOMFuNet model, which integrates complex components such as Graph Convolutional Networks (GCNs) and Transformers, presents a formidable challenge, we can theoretically analyze how its core innovation—the Label Confidence Learning Module (LCLM)—positively influences the model’s generalization capabilities and prediction margin. The LCLM is designed to enhance prediction reliability by promoting orthogonality among the output class probability vectors. This objective is closely related to classical Margin Theory and principles of complexity control in learning theory.

Algorithm 1 Stochastic Training Algorithm for GOMFuNet

Require: Multimodal dataset D, Initial GOMFuNet parameters ${\mathsf{\Theta }}_0$, Regularization hyperparameter $\lambda$, Learning rate $\eta$, Optimizer state $S_0$, Max epochs E, Batch size $B_{size}$. Ensure: Optimized GOMFuNet parameters $\mathsf{\Theta }$. 1: Initialize $\mathsf{\Theta }\leftarrow {\mathsf{\Theta }}_0$, $S\leftarrow S_0$. 2: for epoch = 1 to E do 3:     Shuffle dataset D. 4:     for each batch $B={\left\{(d_{i,1},\dots ,d_{i,M},y_i)\right\}}_{i\in {\mathcal{I}}_B}$ from D do 5:         // Forward Pass: Compute prediction P_B = f_Theta(B) 6:         Compute ${\mathbf{X}}_{m,B}^{\prime }$ for all m. 7:         Compute ${\widehat{\mathbf{H}}}_{m,B}$ via GCN propagation. 8:         Compute ${\mathbf{H}}_{\mathrm{fused},B}$ via concatenation and Transformer aggregation. 9:         Compute ${\mathbf{P}}_B=\mathrm{softmax}({\mathbf{H}}_{\mathrm{fused},B}{\mathbf{W}}_{out}+{\mathbf{b}}_{out})$. 10:        // Loss Computation 11:        Compute ${\mathcal{L}}_{\mathrm{Task},B}=\mathrm{TaskLoss}({\mathbf{P}}_B,y_B)$. 12:        Compute ${\mathcal{L}}_{\mathrm{LCLM},B}$. 13:        Compute ${\mathcal{L}}_{\mathrm{Total},B}={\mathcal{L}}_{\mathrm{Task},B}+\lambda {\mathcal{L}}_{\mathrm{LCLM},B}$. 14:        // Backward Pass and Update 15:        Compute gradient $g_t={\nabla }_{\mathsf{\Theta }}{\mathcal{L}}_{\mathrm{Total},B}$. 16:        Update parameters and optimizer state: $(\mathsf{\Theta },S)\leftarrow \mathrm{OptimizerUpdate}(\mathsf{\Theta },S,g_t,\eta )$. 17:    end for 18:    // Optional: Validation, Learning Rate Decay, Early Stopping 19: end for 20: Return Optimized parameters $\mathsf{\Theta }$.

3.5.1. LCLM, Output Space Geometry, and Class Margin

The LCLM loss function, ${\mathcal{L}}_{\mathrm{LCLM}}(\mathbf{P};{\tau }_L)$, as defined in Equation (8), aims to minimize the cosine similarity between the normalized probability vectors ${\widehat{\mathbf{p}}}_c$ and ${\widehat{\mathbf{p}}}_k$ for distinct classes $c\ne k$. This can be expressed as striving for

$$\langle {\widehat{\mathbf{p}}}_c,{\widehat{\mathbf{p}}}_k\rangle \to 0\mathrm{for}c\ne k.$$

(11)

Geometrically, this is equivalent to encouraging the average probability response directions, represented by these vectors in an N-dimensional space (where N is the batch size), to become mutually orthogonal for different classes.

This pursuit of orthogonality can be interpreted as a form of margin maximization at the level of class probability distributions. In an idealized scenario, if the probability vectors ${\mathbf{p}}_c$ and ${\mathbf{p}}_k$ (representing the model’s probabilistic assignment to class c and k over a batch, or in expectation, over the entire data distribution) are perfectly orthogonal for any two distinct classes, it implies that, statistically, the patterns leading to a high probability assignment for class c are significantly different from those for class k. Such distinctiveness helps the model generate less ambiguous and more definitive class assignments for each input sample.

For an individual sample ${\mathbf{x}}_i$, its predicted probability distribution is $\mathbf{p}\left({\mathbf{x}}_i\right)=(P(y=1|{\mathbf{x}}_i),\dots ,P(y=C|{\mathbf{x}}_i))$. If the model learns class-specific “signatures” (manifested through ${\mathbf{p}}_c$) that are globally orthogonal, then a specific sample ${\mathbf{x}}_i$ is more likely to resonate strongly (i.e., yield a high probability) with only one (or a few closely related) class signature, while maintaining a low similarity (i.e., low probability) with others. This indirectly promotes the “margin” for individual sample predictions. For instance, it encourages a larger difference between the predicted probability of the true class $y_i$, $P(y=y_i|{\mathbf{x}}_i)$, and the highest predicted probability of any incorrect class $c\ne y_i$:

$${\mathrm{Margin}}_i=P(y=y_i|{\mathbf{x}}_i)-\underset{c\ne y_i}{max}P(y=c|{\mathbf{x}}_i).$$

(12)

LCLM’s objective contributes to increasing this margin on average.

3.5.2. LCLM as a Regularizer and Learning Theory

From a learning theory perspective, ${\mathcal{L}}_{\mathrm{LCLM}}$ acts as a regularizer that imposes structural constraints on the model’s output. According to Occam’s Razor and statistical learning theory, controlling model complexity (e.g., by limiting the capacity of the hypothesis space or enforcing solution smoothness through regularization) is crucial for improving generalization, i.e., reducing the gap between empirical risk and expected risk.

Classical generalization error bounds, such as those based on Rademacher complexity, often take the following form:

$$\mathbb{E}\left[\mathcal{L}\left(f\right)\right]\le \widehat{\mathbb{E}}\left[\mathcal{L}\left(f\right)\right]+2{\mathcal{R}}_N\left(\mathcal{F}\right)+\mathrm{complexity}\text{\_}\mathrm{term}$$

(13)

where $\mathcal{L}\left(f\right)$ is the loss function, ${\mathcal{R}}_N\left(\mathcal{F}\right)$ is the empirical Rademacher complexity of the function class $\mathcal{F}$ on a sample of size N. It is well established that classifiers with larger margins (e.g., Support Vector Machines) often correspond to lower Rademacher complexity or VC dimension.

By enforcing orthogonality among output probability vectors, LCLM guides the model towards learning a solution that is “structurally simpler” or exhibits “greater class separation”. Specifically, it penalizes model parameter configurations that lead to highly correlated probability distributions for different classes. This prior knowledge imposed on the output structure can be viewed as restricting the effective hypothesis space, biasing it towards functions that produce high-confidence, highly separable predictions. While we do not explicitly compute the VC dimension or Rademacher complexity of GOMFuNet (which is inherently complex due to its deep GCN and Transformer architecture), the introduction of LCLM aligns with the fundamental principle of improving generalization by increasing the margin or imposing structural constraints.

Furthermore, the orthogonality encouraged by LCLM among class probability vectors is analogous to a feature transformation that renders data points from different classes more linearly separable in the transformed space. If we consider each class probability vector ${\mathbf{p}}_c$ as a form of “average signature” for that class in the “sample space”, LCLM pushes these signatures to be as distinct as possible. This distinctiveness allows decision boundaries to be placed in sparser regions of the output space, thereby enhancing robustness in predictions for unseen samples, which is consistent with the idea of improving generalization margin.

3.5.3. Error Probability and Its Relation to Margin

In classification tasks, the error probability is intrinsically linked to the margin of the decision boundary. For a sample $(\mathbf{x},y_{\mathrm{true}})$, if the model predicts $P\left(y_{\mathrm{true}}\right|\mathbf{x})$ to be substantially greater than $P\left(y_j\right|\mathbf{x})$ for all $j\ne y_{\mathrm{true}}$ (i.e., it possesses a large prediction margin as in Equation (12)), the probability of misclassifying this sample is low. LCLM, by enforcing separation between class-level probability distributions on a batch basis, contributes to enhancing this prediction margin in an average sense. When the model can correctly classify most training samples with high confidence (large margin), its error probability on the test set also tends to be lower.

4. Case Study: Educational Performance Prediction

To evaluate the practical effectiveness and demonstrate the applicability of the proposed GOMFuNet framework, we conduct a case study in the domain of educational performance prediction. This domain provides a challenging real-world scenario involving heterogeneous multimodal data where prediction reliability is valuable.

4.1. Dataset and Preprocessing

The data for this case study were sourced from two cohorts of students (totaling 472 students: 243 first cohort, 229 s) in an English course over two academic years. The dataset includes classroom performance records derived from video analysis, midterm online exam scores (including written scores, text essays, and audio recordings), and final exam scores. The objective is to predict students’ final English grades, framed as both a binary classification task (Pass/Fail, based on a score threshold of 60) and a regression task (predicting the specific percentage score). Ethical considerations were addressed; all participating students were informed and consented to the use of their anonymized learning data for research analysis.

Classroom performance data were extracted from video recordings using a combination of automated scripts (OpenCV) and manual annotation for metrics such as attendance, seating position (row/column, 0 if absent), proportion of time looking at the instructor, estimated note-taking time proportion, and participation count (hand-raising/speaking instances). Midterm data included structured written test scores (0–100), one-minute oral recordings (WAV format), and written assignments (TXT format). Final exam data, comprising pass/fail status (0/1) and specific scores (0–100), served as the prediction targets.

Preprocessing involved anonymization, standardization, and cleaning. Structured numerical features were normalized to the [0–1] range using Min–Max scaling. Audio (WAV) data underwent noise reduction (spectral subtraction) and segmentation using Voice Activity Detection (WebRTC VAD). Text (TXT) data were cleaned, tokenized using the WordPiece tokenizer (‘bert-base-uncased’), formatted with special tokens ([CLS], [SEP], [PAD]), and converted to fixed-length sequences with attention masks. Missing values were handled appropriately (e.g., removed if critical). The distribution of final exam scores in the combined dataset (N = 472) is shown in

Table 1. Distribution of students’ final exam scores in the case study dataset (N = 472).

Classification	Score Range	Number of Students
Fail	<60	56
Pass	60–69	66
Pass	70–79	110
Pass	80–89	157
Pass	90–99	75
Pass	100	8
Total		472

.

4.2. Experimental Setup and Evaluation Metrics

Experiments were conducted using PyTorch 1.10.0 on a system with an NVIDIA Tesla V100 GPU. The dataset (N = 472) was split into training (80%, N = 378) and testing (20%, N = 94) sets, stratified by the pass/fail status. Five-fold cross-validation on the training set was used for hyperparameter tuning (including the LCLM weight $\lambda$, GCN layers, Transformer parameters, learning rate, batch size). The final reported results are averaged over 5 independent runs with different random seeds on the held-out test set, after training the model on the full 80% training data using the best hyperparameters found. Key hyperparameters for the GOMFuNet core are detailed in

Table 2. Key architectural details and hyperparameters of the GOMFuNet core modules.

Component	Parameter	Value
Feature Encoders (Output Dims)	Structured ($d_s^{\prime }$)	32
	Text ($d_t^{\prime }$)	768
	Audio ($d_a^{\prime }$)	128
CLFM: Graph Construction	Method	kNN + Gaussian Kernel
CLFM: GCN	Layers ($L_g$)	2
	Hidden Dimension	128
	Output Dimension (per mod, $d_m^{\prime \prime }$)	64
	Activation	ReLU
CLFM: Transformer Aggregator	Layers	2
	Heads	4
	Model Dimension ($\sum d_m^{″}$)	192 (3 × 64)
	Feedforward Inner Dimension	256
	Dropout	0.2
	Output Dimension ($d_{\mathrm{fused}}$)	192
LCLM	Weighting Factor ($\lambda$)	[Specify best lambda value]
	Temperature (${\tau }_L$)	1.0
Training	Optimizer	Adam
	Initial Learning Rate	0.001
	Batch Size	32
	Epochs	35 (with Early Stopping)

, while encoder details are given in Appendix A.

The Adam optimizer was used with an initial learning rate of 0.001, standard betas, and epsilon = 1 × 10⁻⁸. A ‘ReduceLROnPlateau’ learning rate scheduler (patience = 5, factor = 0.1) and early stopping (patience = 10 based on validation loss) were employed. The batch size was 32. The LCLM weighting factor $\lambda$ was tuned via cross-validation (tested values e.g., [0.01, 0.1, 1.0]) and set to [Specify best lambda value here].

Performance was evaluated using standard metrics. For the classification task (Pass/Fail): Accuracy, F1-score, and Area Under the ROC Curve (ROC-AUC). For the regression task (Predicting exact score): Mean Squared Error (MSE), Mean Absolute Error (MAE), and Coefficient of Determination (R-squared, $R^2$). The formulas for these metrics are standard and defined as follows: Accuracy $={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$; F1-score $=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$; ROC-AUC is the area under the Receiver Operating Characteristic curve; MSE $={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2$; MAE $={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n|{\widehat{y}}_i-y_i|$; $R^2=1-{\displaystyle \frac{{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}}$. TP, TN, FP, and FN represent True Positives, True Negatives, False Positives, and False Negatives; n is the number of test samples; $y_i$ is the true value; ${\widehat{y}}_i$ is the predicted value; and $\overline{y}$ is the mean of the true values.

4.3. Baselines and Comparison Methods

To assess the effectiveness of GOMFuNet, we compare it against two sets of baseline methods on the case study data:

• Single-Modality Models: Models trained on each data modality (structured, text, audio) independently using appropriate machine learning algorithms (Decision Tree, Random Forest, SVM, NN for structured; LSTM, CNN, LSTM + CNN for audio; Naive Bayes, RNN, BERT, BERT + MHA for text). This establishes the performance level achievable without multimodal fusion.
• State-of-the-Art Multimodal Models: We compare GOMFuNet against several existing advanced multimodal fusion techniques reported in the literature, implemented on our dataset: ATV [41], MsaCNN [42], and Transmodality [43]. This comparison highlights GOMFuNet’s performance relative to other fusion paradigms.

Additionally, we conduct ablation studies by removing key components of GOMFuNet (CLFM, LCLM) or individual modalities to understand their specific contributions.

5. Results and Analysis

This section presents the empirical results of the case study evaluating the proposed GOMFuNet framework on the task of predicting student English course performance.

5.1. Comparison with Single-Modality Baselines

Table 3. Performance comparison of single-modality models and the proposed GOMFuNet on classification and regression tasks (case study dataset).

Model	Modality	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
Decision Tree	Structured	66.05%	0.63	0.71	0.0348	0.1535	62.95%
Random Forest	Structured	69.12%	0.65	0.74	0.0331	0.1460	66.81%
Support Vector Machine	Structured	67.98%	0.64	0.73	0.0340	0.1501	64.92%
Neural Network	Structured	71.15%	0.67	0.76	0.0319	0.1405	69.33%
LSTM	Audio	60.33%	0.58	0.66	0.0405	0.1760	53.81%
CNN	Audio	62.01%	0.60	0.68	0.0391	0.1695	56.24%
LSTM + CNN	Audio	63.42%	0.61	0.69	0.0378	0.1632	58.44%
Naive Bayes	Text	73.02%	0.70	0.79	0.0308	0.1340	70.95%
BERT	Text	75.88%	0.73	0.81	0.0289	0.1265	74.89%
RNN	Text	73.91%	0.71	0.80	0.0299	0.1301	72.98%
BERT + MHA	Text	76.73%	0.74	0.83	0.0280	0.1240	76.31%
GOMFuNet (Proposed)	Multimodal	90.17%	0.88	0.92	0.0205	0.0971	88.03%

compares the performance of GOMFuNet against models trained on single data modalities. The results clearly show that GOMFuNet, leveraging the fusion of structured, text, and audio data, significantly outperforms all single-modality baselines across both classification (Accuracy, F1, AUC) and regression ($R^2$, MSE, MAE) tasks. For instance, the best single-modality model (BERT + MHA on Text) achieves 76.73% accuracy and 76.31% $R^2$, whereas GOMFuNet reaches 90.17% accuracy and 88.03% $R^2$. This substantial improvement underscores the necessity of multimodal integration to capture the complex factors influencing student performance, a task effectively addressed by the GOMFuNet framework. The limitations of relying on any single data source are evident, highlighting the value captured by GOMFuNet’s fusion mechanism.

5.2. Comparison with State-of-the-Art Multimodal Baselines

Table 4. Performance comparison of GOMFuNet with other advanced multimodal models (case study dataset).

Model	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
ATV [41]	87.05%	0.84	0.89	0.0245	0.1055	85.88%
MsaCNN [42]	86.31%	0.83	0.88	0.0256	0.1091	84.41%
Transmodality [43]	85.19%	0.82	0.87	0.0268	0.1130	82.74%
GOMFuNet (Proposed)	90.17%	0.88	0.92	0.0205	0.0971	88.03%

compares GOMFuNet with other advanced multimodal fusion models implemented on our case study dataset. While methods like ATV, MsaCNN, and Transmodality achieve respectable performance, GOMFuNet consistently outperforms them across all reported metrics. For example, the best competitor (ATV) achieves 87.05% accuracy and 85.88% $R^2$, compared to GOMFuNet’s 90.17% and 88.03%, respectively. This superiority suggests that GOMFuNet’s specific mathematical design—combining geometric fusion in the label space via CLFM with explicit reliability optimization via LCLM—provides tangible advantages over alternative fusion strategies in handling the complexities of this multimodal dataset.

5.3. Impact of LCLM (Loss Function Comparison)

To isolate the contribution of the novel LCLM component, we compared the performance of the GOMFuNet architecture when trained using the combined loss versus using only standard loss functions (Cross-Entropy for classification, MSE adapted, and Focal Loss as an advanced baseline).

Table 5. Performance comparison of GOMFuNet using different loss function configurations.

Loss Configuration (Within GOMFuNet)	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
CLFM + Cross-Entropy Loss	87.88%	0.85	0.90	0.0229	0.1028	86.61%
CLFM + Mean Squared Error Loss	89.01%	0.86	0.91	0.0220	0.1005	87.32%
CLFM + Focal Loss [31]	89.52%	0.87	0.91	0.0214	0.0989	87.88%
CLFM + LCLM + Task Loss (Proposed)	90.17%	0.88	0.92	0.0205	0.0971	88.03%

shows these results. Using LCLM yields the best performance across all metrics, surpassing even Focal Loss (which is designed for imbalance but does not optimize output geometry). For instance, accuracy improves from 89.52% (Focal Loss) to 90.17% (LCLM), and $R^2$ improves from 87.88% to 88.03%. This directly demonstrates the empirical benefit of the LCLM’s mathematical objective of enforcing label orthogonality for enhancing prediction accuracy and reliability, compared to standard methods focusing solely on label matching.

5.4. Ablation Studies

We performed ablation studies to understand the contribution of individual modalities and core GOMFuNet modules.

Table 6. Modality ablation study on GOMFuNet performance.

Structured	Text	Audio	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
✔	✔	✔	90.17%	0.88	0.92	0.0205	0.0971	88.03%
×	✔	✔	87.41%	0.85	0.89	0.0242	0.1050	85.11%
✔	×	✔	86.05%	0.83	0.87	0.0260	0.1105	83.49%
✔	✔	×	88.12%	0.86	0.90	0.0235	0.1029	86.15%

shows the impact of removing each modality. Removing any single modality leads to a performance drop, confirming that each data stream provides valuable complementary information captured by GOMFuNet. Text data appear most influential in this case study, followed by structured data, and then audio data, but the best performance is achieved only when all three are fused by GOMFuNet.

Table 7. GOMFuNet module ablation study on overall model performance.

CLFM	LCLM	Accuracy	F1-Score	ROC-AUC	MSE	MAE	R2
✔	✔	90.17%	0.88	0.92	0.0205	0.0971	88.03%
×	✔	88.03%	0.86	0.90	0.0236	0.1030	85.99%
✔	×	89.52%	0.87	0.91	0.0214	0.0989	87.88%
×	×	85.39%	0.83	0.88	0.0271	0.1118	82.15%

shows the impact of removing the core CLFM or LCLM modules. Removing CLFM (implying simpler fusion before LCLM and task loss) causes a significant drop (Accuracy 88.03%, $R^2$ 85.99%), highlighting the importance of the geometric fusion approach in CLFM. Removing LCLM (using CLFM with only task loss, akin to the Focal Loss result in Table 5) also degrades performance (Accuracy 89.52%, $R^2$ 87.88%), confirming the contribution of the LCLM’s explicit reliability optimization. Removing both modules leads to the worst performance, emphasizing the synergistic benefit of GOMFuNet’s combined mathematical design.

5.5. Visualization of Fusion Effect

To qualitatively assess the impact of the CLFM’s geometric fusion, we visualize the high-dimensional feature representations using t-SNE before and after the CLFM module. Figure 1 shows the embeddings colored by final course status (Pass/Fail) and numerical score. Before fusion (left column), features from simple concatenation show poor separation between classes and little structure related to scores. After fusion by CLFM (right column), the embeddings exhibit much clearer separation between Pass and Fail clusters and a more discernible structure corresponding to numerical scores. This visualization provides qualitative evidence that CLFM effectively integrates multimodal information into a more mathematically meaningful and discriminative feature space compared to naive concatenation, supporting its role in the GOMFuNet framework.

5.6. Interpretability via Feature Importance Analysis

To gain insights into the decision-making process of the GOMFuNet model within the case study context, particularly regarding the contribution of readily interpretable input features, we employed the SHAP (SHapley Additive exPlanations) framework. Figure 2 presents the SHAP summary plot (beeswarm) focusing on the impact of the structured input features (derived from classroom video analysis and exam scores) on the model’s prediction towards the ‘Pass’ outcome. Each point represents a student sample from the test set, where the color indicates the feature’s value (red = high, blue = low) and the horizontal position reflects the SHAP value, quantifying the feature’s contribution to shifting the prediction probability. This analysis reveals that ‘Midterm Score’ exerts the most substantial influence, with higher scores strongly driving predictions towards ‘Pass’. Other educationally relevant factors like ‘Participation Count’ and ‘Attendance Rate’ also emerge as significant positive predictors, aligning with pedagogical intuition. Conversely, features such as ‘Seat Row’ exhibit a smaller, context-dependent impact. This interpretability analysis enhances confidence in the model by demonstrating that GOMFuNet learns meaningful relationships from the input data, leveraging key indicators of student performance in a discernible manner.

5.7. Prediction Accuracy Analysis (ROC and Scatter Plots)

Figure 3 presents the ROC curve for the classification task and the scatter plot of actual versus predicted scores for the regression task, both obtained using the full GOMFuNet model. The high ROC-AUC value of 0.92 indicates strong discriminative ability between Pass and Fail classes. The scatter plot shows predicted scores closely aligning with the actual scores along the diagonal, demonstrating good predictive accuracy for the regression task, reflected also in the high $R^2$ value of 88.03%. Some deviations exist for outlier cases, but the overall trend confirms GOMFuNet’s effectiveness in both prediction tasks within the case study.

6. Discussion

This paper introduced GOMFuNet, a novel mathematical framework for multimodal data fusion designed to enhance prediction reliability. The core innovations lie in the synergistic combination of geometric deep learning for fusion in the label space (CLFM) and an explicit optimization objective based on output vector orthogonality for confidence enhancement (LCLM). The empirical results from the educational performance prediction case study provide strong validation for the proposed mathematical concepts. GOMFuNet significantly outperformed both single-modality baselines and state-of-the-art multimodal fusion methods, demonstrating the practical benefits of its design. The ablation studies confirmed the necessity of both the CLFM’s geometric fusion approach and the LCLM’s reliability optimization for achieving peak performance. Furthermore, the analysis of confidence calibration and robustness to noise lends empirical support to the mathematical claims that GOMFuNet, particularly through LCLM, improves prediction reliability and stability beyond what is typically achieved with standard loss functions. The visualization of the fused feature space also qualitatively supports the effectiveness of CLFM’s fusion strategy.

From a mathematical perspective, GOMFuNet offers several advantages. The CLFM’s use of GCNs in the label space allows explicit modeling of inter-sample relationships at a high semantic level, potentially offering better robustness to low-level feature noise and inconsistencies compared to feature-level fusion. The LCLM introduces a novel way to regularize the output space directly, promoting class separability and confidence through a geometrically motivated orthogonality constraint. This explicit focus on the output structure complements traditional loss functions that focus on point-wise prediction accuracy. The synergy between preserving input structure (via GCN) and structuring the output space (via LCLM) appears crucial to GOMFuNet’s success.

However, the proposed framework also has limitations. Mathematically, we have provided justifications and empirical validations, but formal theoretical guarantees on error probability bounds or convergence properties of the LCLM loss remain subjects for future investigation. The LCLM loss, like many deep learning objectives, is likely non-convex, and its optimization dynamics warrant further study. The specific graph construction method involves hyperparameters (like k in kNN) that might require tuning. Computationally, GOMFuNet involves multiple components (encoders, GCNs, Transformer, LCLM calculation), potentially leading to higher computational cost compared to simpler models, although this was manageable for our case study dataset size. From an application perspective, the generalizability of GOMFuNet’s performance benefits needs to be validated across diverse multimodal datasets from different domains beyond education. The effectiveness might depend on the nature of the modalities and the underlying relationships between them.

The implications of this work extend to the broader fields of multimodal learning and trustworthy AI. It highlights the potential of integrating principles from geometric deep learning with novel optimization objectives that directly target prediction reliability. GOMFuNet provides a concrete example of how moving beyond standard accuracy-focused training can lead to more robust and confidence-aware models, which is critical for high-stakes applications.

Future work should focus on several directions. Developing theoretical analyses of LCLM’s properties (e.g., convergence, relationship to generalization bounds, impact on calibration) would strengthen its mathematical foundations. Exploring alternative geometric structures or propagation mechanisms within CLFM could yield further improvements. Investigating adaptive methods for setting the LCLM weight $\lambda$ or extending the LCLM concept to other tasks like structured prediction or unsupervised learning presents interesting avenues. Applying and adapting GOMFuNet to other challenging multimodal problems (e.g., medical diagnosis, affective computing, robotics) will be crucial for assessing its broader utility and identifying necessary modifications. While the educational case study demonstrated GOMFuNet’s potential, further refinement for specific educational interventions, perhaps by incorporating domain knowledge or enhancing interpretability (e.g., using techniques like SHAP as explored in preliminary analyses, potentially detailed in Appendix A, remains a secondary but valuable direction.

Empirical Validation of Prediction Reliability via Calibration Analysis

A critical aspect of trustworthy prediction systems lies in the statistical consistency between the model’s expressed confidence and its empirical accuracy. To rigorously evaluate the mathematical property of enhanced reliability fostered by the LCLM component, we analyze the confidence calibration of GOMFuNet’s probabilistic outputs. A model f is considered perfectly calibrated if, for any predicted probability value $p\in [0,1]$, the conditional expectation of the true label Y given the predicted probability $f\left(X\right)=p$ equals p—i.e., $\mathbb{E}\left[Y\right|f\left(X\right)=p]=p$. Deviations from this ideal condition indicate miscalibration, potentially leading to unreliable decision-making based on the model’s confidence estimates.

We quantify calibration performance using the Expected Calibration Error (ECE), a standard metric defined as the weighted average of the absolute difference between accuracy and confidence across K pre-defined probability bins $B_k$:

$$\mathrm{ECE}=\sum _{k=1}^K{\displaystyle \frac{|B_k|}{N}}\left|\mathrm{acc}\left(B_k\right)-\mathrm{conf}\left(B_k\right)\right|$$

(14)

where N is the total number of samples, $|B_k|$ is the number of samples whose predicted confidence falls into bin $B_k$, $\mathrm{acc}\left(B_k\right)$ is the empirical accuracy of samples in bin $B_k$, and $\mathrm{conf}\left(B_k\right)$ is the average predicted confidence for samples in bin $B_k$. A lower ECE signifies superior calibration. Additionally, we utilize Reliability Diagrams, which visually plot $\mathrm{acc}\left(B_k\right)$ against $\mathrm{conf}\left(B_k\right)$ for each bin k; perfect calibration corresponds to points lying on the diagonal line $y=x$.

Figure 4 presents the reliability diagrams comparing the GOMFuNet architecture trained with the standard task loss (${\mathcal{L}}_{\mathrm{Task}}$) alone versus the proposed configuration utilizing the combined loss (${\mathcal{L}}_{\mathrm{Task}}+\lambda {\mathcal{L}}_{\mathrm{LCLM}}$). The corresponding ECE values, computed using $K=10$ bins, are reported in

Table 8. Expected Calibration Error (ECE) comparison.

Model Configuration	ECE (%)
GOMFuNet (CLFM + Standard Loss)	6.15
GOMFuNet (CLFM + LCLM + Standard Loss)	2.83

. A marked improvement in calibration is observed when incorporating the LCLM objective. GOMFuNet trained with LCLM achieves a significantly lower ECE of 2.83%, compared to an ECE of 6.15% for the baseline model trained using only the standard cross-entropy loss. Furthermore, the reliability diagram associated with the LCLM-enhanced GOMFuNet exhibits points clustering more closely around the diagonal of perfect calibration, indicating a stronger alignment between predicted confidence levels and empirical probabilities of correctness across the spectrum of confidence values.

This empirical evidence provides strong quantitative support for the hypothesis that the LCLM loss, by enforcing orthogonality among class prediction vectors, directly contributes to producing mathematically more meaningful and statistically reliable confidence estimates. The substantial reduction in ECE validates LCLM’s effectiveness in optimizing the reliability structure of the predictions, moving beyond mere accuracy improvements towards generating more trustworthy probabilistic outputs.

7. Conclusions

This paper addressed the mathematical challenges of fusing heterogeneous multimodal data while ensuring the reliability of predictions. We proposed the Geometric Orthogonal Multimodal Fusion Network (GOMFuNet), a novel framework characterized by two key mathematical innovations: (1) geometric fusion in a high-level label space using Graph Convolutional Networks (CLFM) to preserve topological structure and handle inconsistencies, and (2) explicit prediction reliability enhancement via a novel Label Confidence Learning Module (LCLM) that enforces orthogonality among output class probability vectors.

Through mathematical justification and rigorous empirical evaluation, including confidence calibration, robustness analysis, and a detailed case study on educational performance prediction, we demonstrated that GOMFuNet provides a principled and effective approach to multimodal learning. GOMFuNet significantly outperformed single-modality baselines and existing state-of-the-art multimodal fusion techniques in terms of both prediction accuracy and reliability metrics. The ablation studies confirmed the synergistic contribution of both the CLFM and LCLM modules. Our findings highlight the value of integrating geometric deep learning principles with direct optimization of output space structure for building more trustworthy AI systems. GOMFuNet offers a promising direction for future research in reliable multimodal data analysis and its application to complex real-world problems. Future mathematical work includes deriving theoretical guarantees for LCLM and exploring extensions of the GOMFuNet framework.