A CNN-ViT Hybrid Architecture Res101-MViT-Ens for Accurate and Lightweight Automated Ocular Disease Diagnosis

Abstract

Automated ocular disease diagnosis faces critical challenges including insufficient diagnostic precision, local–global feature imbalance, rigid feature fusion, weak cross-domain generalization, and difficult lightweight deployment. This study aims to develop a high-performance, generalizable, and deployable hybrid deep learning architecture for accurate multi-class ocular disease diagnosis. We propose the Res101-MViT-Ens hybrid architecture, which fuses ResNet101 for local fine-grained feature extraction and MobileViT-XXS for global contextual modeling via an end-to-end dynamic learnable weight fusion mechanism, with class-balanced sampling and medically adaptive augmentation for data preprocessing. The model is validated on the ODIR-5K dataset and cross-evaluated on three heterogeneous datasets (MESSIDOR-2, Kaggle DR, EyePACS). It achieves 99.44% accuracy, a 99.41% F1-score, and 99.32% Kappa on ODIR-5K, with a 99.46% average cross-dataset accuracy, outperforming state-of-the-art models. With 54 M parameters and 42.6 ms per-image inference latency on the Snapdragon 8 Gen2 edge module (Qualcomm Technologies, Inc., San Diego, CA, USA), it outperforms mainstream edge architectures. This proposed architecture achieves state-of-the-art diagnostic precision; balances accuracy, generalization and practicality; and is suitable for lightweight grassroots deployment in ocular disease screening.

1. Introduction

Ocular diseases constitute a major global public health issue, as Glaucoma, Diabetic Retinopathy, and Age-related Macular Degeneration can lead to irreversible visual impairment or even blindness without timely diagnosis and intervention. The automated classification of fundus images, which visualize pathological changes in retinal blood vessels, the macula, and other key ocular structures, has become the core technical direction to break the efficiency bottleneck of manual diagnosis, alleviate the uneven distribution of ophthalmic medical resources, and expand the coverage of grassroots screening [1]. The ODIR-5K dataset, covering eight categories of ocular diseases with complete clinical annotations and a high heterogeneity of image sources from 5000 patients, has been established as the core benchmark for multi-class classification tasks in ophthalmic artificial intelligence, providing a standardized validation basis for generalizable diagnostic models [2].

Existing CNN-ViT hybrid models for fundus image-based ocular disease diagnosis have become the mainstream research paradigm, as they attempt to integrate the complementary advantages of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) [3,4,5]. CNNs excel at automatically identifying intricate pathological motifs in medical images, and the classic deep CNN architecture ResNet101 mitigates gradient vanishing in deep networks via residual connections, enabling the stable extraction of local fine-grained features such as retinal vascular morphology and microhemorrhages in ocular disease classification [6,7,8]. ViTs and their lightweight variants (e.g., MobileViT-XXS) rely on the global self-attention mechanism to model long-range contextual relationships, such as the overall distribution of retinal lesions and spatial correlations of structural abnormalities, which compensate for the inherent limitations of CNNs [9,10]. However, these existing hybrid models have not yet resolved the core contradiction of local–global feature imbalance, which we formally define as follows: a phenomenon in which fundus images contain both micro-scale local pathological features (e.g., micro-exudates, dot hemorrhages) and macro-scale global structural abnormalities (e.g., extensive retinal edema), yet existing models fail to achieve adaptive capture and weight allocation of these local and global dual features according to the pathological characteristics of different ocular diseases, ultimately resulting in incomplete and non-targeted feature extraction. Although researchers have attempted to alleviate this issue through transfer learning, data augmentation, and fixed-weight fusion [11,12], these strategies cannot fundamentally address the adaptive matching between multi-scale pathological features and model feature extraction capabilities.

Rather than a universally unsolved problem in the entire field of automated ocular disease diagnosis, the remaining technical bottlenecks are specifically concentrated in the following four core limitations of existing CNN-ViT hybrid diagnostic models: (1) Unresolved local–global feature imbalance: While existing hybrid models achieve a preliminary fusion of CNN and ViT features, they still cannot achieve balanced and adaptive capture of micro-lesion details and macro-structural correlations, leading to insufficient feature representation for multi-scale pathological changes in fundus images. (2) Rigid feature integration strategies: Most existing hybrid models, including the Doctor AI framework proposed by Thinh et al. [3] and HybridVisionNet developed by Kılıç et al. [4], adopt fixed-weight fusion, which cannot adaptively adjust the contribution ratio of the CNN and ViT sub-models according to disease types and image feature distributions, resulting in insufficient flexibility in multi-class classification tasks. (3) Weak cross-domain generalization performance: Differences in imaging equipment, shooting conditions, and annotation standards are prone to cause “domain shift”, and most existing hybrid models are trained and validated on a single dataset, leading to significant performance degradation on heterogeneous data such as low-resolution images from grassroots clinical settings, which cannot meet the requirements of multi-scenario clinical application. (4) Unresolvable conflict between high performance and lightweight deployment: Most high-performance hybrid models such as EffViT-B0 rely on complex architectures with up to 78.5 M parameters, exceeding the computing threshold of lightweight hardware in grassroots medical settings, while simplified lightweight hybrid models often suffer from insufficient feature extraction capability and reduced diagnostic accuracy, and to date, no existing hybrid model has formed an integrated solution that simultaneously balances multi-scale feature capture, adaptive fusion, strong cross-domain generalization, and lightweight deployment.

To address the above four core limitations of existing hybrid models, this study proposes a learnable weight-based hybrid integrated architecture, Res101-MViT-Ens (full name: ResNet101-MobileViT-XXS Ensemble), with targeted designs from four dimensions: architecture fusion, feature fusion mechanism, data processing, and lightweight design. Specifically, the architecture adopts a parallel dual-branch structure integrating ResNet101 and MobileViT-XXS to fully leverage their complementary advantages in local–global feature extraction, resolving the local–global feature imbalance limitation; it designs an end-to-end dynamic learnable weight fusion mechanism with softmax normalization to replace traditional fixed-weight integration, adaptively optimizing the feature contribution ratio of the two sub-models to solve the rigidity of fusion strategies; it adopts class-balanced sampling combined with medically adaptive data augmentation, and conducts systematic cross-dataset validation on MESSIDOR-2, Kaggle DR and EyePACS to enhance the model’s robustness to heterogeneous data; meanwhile, it controls the total parameter count within 54 M (only 63% of ViT-Base) with single-image inference latency below 50 ms by virtue of the lightweight property of MobileViT-XXS, effectively resolving the conflict between high diagnostic accuracy and lightweight deployment.

1.1. Main Contributions

The main methodological innovations and academic contributions of this study are summarized as follows, with each contribution directly targeting the core limitations of existing hybrid models identified in the introduction:

• To address the fundamental challenge of local–global feature imbalance in fundus image analysis, we propose the Res101-MViT-Ens hybrid architecture with a parallel dual-branch design. Unlike existing architectures that only achieve preliminary concatenation and fusion of CNN and ViT features, this framework innovatively integrates ResNet101 and MobileViT-XXS to fully exploit their inherent complementary advantages: the deep residual structure of ResNet101 enables the fine-grained capture of micro-pathological features such as micro-exudates and dot hemorrhages, while the lightweight global attention mechanism of MobileViT-XXS facilitates the cross-regional contextual modeling of overall lesion distribution patterns. This design fundamentally breaks through the limitation that existing models cannot achieve a balanced and adaptive extraction of multi-scale pathological features, and the effectiveness of this architectural paradigm is verified by a classification accuracy of 99.44% on the ODIR-5K dataset, which is 7.11% higher than the single CNN branch ResNet101 and 7.32% higher than the single ViT branch MobileViT-XXS.
• To solve the rigid feature integration problem of existing hybrid models, we design an end-to-end dynamic learnable weight fusion mechanism with softmax normalization constraints, which has three core differentiations from mainstream standard attention, gating mechanisms, and weighted ensemble techniques: (1) Training paradigm difference: it realizes the end-to-end synchronous optimization of weights and dual-branch feature extraction, instead of the post-fusion with fixed weights in traditional weighted ensembles that cannot be adjusted during training. (2) Weighting dimension difference: it realizes disease feature-driven global contribution allocation at the sub-model level, rather than the local channel/spatial feature weighting in conventional gating mechanisms (SE/CBAM). (3) Interpretability constraint difference: the softmax normalization ensures the sum of weights is 1, which quantifies the contribution ratio of dual branches (e.g., ResNet101 weight of 0.5419 and MobileViT-XXS weight of 0.4581 on ODIR-5K) and avoids the training instability of unconstrained standard self-attention. This mechanism replaces the traditional fixed-weight integration strategy, adaptively optimizes the contribution ratio of the two sub-models according to disease types and image feature distributions, and systematic ablation experiments quantitatively confirm that this mechanism can increase the model’s F1-score by 5~6% compared with fixed-weight fusion, effectively improving the stability and flexibility of classification performance.
• Targeting the challenge of weak cross-domain generalization in clinical scenarios, we construct a multi-scenario validation system and a medically adaptive data preprocessing pipeline for heterogeneous data. Aiming at the “domain shift” caused by differences in imaging equipment, shooting conditions and annotation standards, we adopt class-balanced sampling and medically adaptive augmentation in the preprocessing stage to enhance the model’s feature robustness, and conduct systematic evaluations on three heterogeneous datasets with distinct imaging protocols and task types: MESSIDOR-2 for single-disease diagnosis, Kaggle DR for lesion grading, and EyePACS for Diabetic Retinopathy detection. The model achieves accuracies of 99.75%, 99.65%, and 98.75% on the three datasets respectively, with an average accuracy of 99.46%, which fully verifies its robustness to diverse clinical imaging scenarios and different diagnostic tasks.
• For the conflict between high diagnostic accuracy and lightweight edge deployment, we build a lightweight and efficient deployment scheme with complete hardware verification and head-to-head comparison against established edge-class architectures. By leveraging the lightweight property of MobileViT-XXS, the total parameter count of Res101-MViT-Ens is controlled within 54 M in native FP32 precision, which is only 63% of ViT-Base, 6.7% lower than the industry-standard edge deployment baseline MobileNetV3-Large, and 12.3% lower than EfficientNet-Lite4, while maintaining state-of-the-art diagnostic accuracy without complex manual feature engineering. We further provide hardware-specific deployment evidence with fully disclosed test conditions: on the Snapdragon 8 Gen2 edge computing module (mainstream for grassroots portable devices), the end-to-end inference latency of a single 224 × 224 fundus image is only 42.6 ms, with a peak memory usage of 128 M and peak computing power demand of 8.0 TOPS, all of which meet the mainstream hardware thresholds of grassroots portable fundus cameras (memory ≤ 256 M, computing power ≤ 10 TOPS). This confirms the model’s natively lightweight property, as it meets edge deployment requirements without any compression or pruning optimization. In addition, the model supports deployment optimization via INT8 quantization: after quantization, the parameter count is reduced to 13.5 M, the inference latency is shortened to 18.7 ms, and the accuracy is only reduced by 0.52 percentage points, which can be directly integrated into mainstream grassroots portable fundus cameras such as Canon CR-2 Plus (Canon Inc., Tokyo, Japan) without additional computing power expansion, further improving its adaptability to ultra-low-power grassroots lightweight hardware.

The architectural paradigm and dynamic fusion mechanism proposed in this study can also provide a reusable methodological reference for multi-scale feature fusion tasks in other medical image analysis scenarios.

1.2. Related Work

To further highlight the innovative value of this study, this section systematically reviews the related work in automated ocular disease classification with a gap-oriented comparative analytical framework. The core proposition of this study is to construct a CNN-ViT hybrid architecture with adaptive weight fusion for accurate multi-class ocular disease diagnosis, and the literature review is carried out from four dimensions—inherent limitations of single-branch architectures, methodological defects of existing hybrid architectures, capability boundaries of adaptive fusion methods, and the balance bottleneck between lightweight deployment and cross-domain generalization—to precisely locate the unmet methodological gaps in the current field.

In terms of the inherent limitations of single-branch architectures in ocular disease diagnosis, CNN-based methods have been the mainstream technical route for early fundus image analysis, relying on their powerful local fine-grained feature extraction capability. Classic deep CNN architectures such as ResNet and DenseNet alleviate the gradient vanishing problem of deep networks through residual or dense connections [8], and have been widely verified in the capture of microvascular lesions and exudates in fundus images. For example, Khan et al. [13] proposed an ocular disease detection scheme based on ensembled deep CNNs, using ResNet50 as the core feature extractor to capture local pathological details such as vascular morphology and micro-lesion characteristics from enhanced fundus images, and the integration of multiple CNN architectures further improved the classification reliability for diverse ocular disease cases; Rasappan et al. [14] also realized efficient multi-disease classification on the ODIR-5K dataset through a customized CNN architecture, giving full play to the advantage of local feature capture of CNNs. However, the inherent limitation of CNNs lies in their limited receptive field, which makes them unable to model cross-regional global contextual dependencies in fundus images, thus restricting their recognition performance for diseases that rely on overall structural judgment (e.g., some cases of Age-related Macular Degeneration), and this core defect cannot be fundamentally solved by simply stacking CNN layers or ensembling multiple CNN models. Vision Transformers (ViTs) and their variants compensate for the above shortcomings of CNNs through the global self-attention mechanism, which can effectively model long-range contextual relationships such as the overall distribution of retinal lesions and spatial correlations of structural abnormalities, and have been widely applied in medical image analysis [15]. Dosovitskiy et al. [9] pioneered the application of ViTs in image classification, and verified the effectiveness of the self-attention mechanism in long-range dependency modeling of visual content. However, standard ViT models have two unavoidable methodological defects in ocular disease diagnosis tasks: first, they usually require pre-training on large-scale datasets, and have relatively weak perception of local micro-lesions in fundus images, resulting in lower accuracy than CNNs when processing subtle pathological changes; second, the large parameter count of standard ViT models leads to high computational costs, which are not conducive to clinical lightweight deployment on edge devices [16,17]. The above comparative analysis shows that single-branch architectures (whether CNNs or ViTs) cannot resolve the core contradiction of local–global feature imbalance in fundus image diagnosis, as they cannot simultaneously achieve the efficient capture of micro-scale local lesion details and macro-scale global pathological correlations, which provides the core motivation for the construction of CNN-ViT hybrid architectures, and is also the primary methodological gap that this study aims to address.

For the methodological defects of existing CNN-ViT hybrid architectures, hybrid architecture methods that integrate the complementary advantages of CNNs and ViTs have become the current research hotspot in multi-class ocular disease diagnosis, and existing studies can be divided into two mainstream paradigms: the serial feature extraction paradigm and parallel fixed-weight fusion paradigm [5,18,19]. The serial paradigm usually uses CNNs as the front-end feature extractor, and then inputs the extracted feature maps into Transformers for global relationship modeling. For example, Abdullah et al. [20] extracted hybrid deep features based on CNNs, and improved the model robustness by combining traditional classifiers, but this paradigm loses the complementary feature-learning capability of the dual branches during the serial transmission process, and cannot achieve a balanced capture of local and global features. The parallel fixed-weight fusion paradigm is the technical route most closely resembling our proposed method, which constructs independent CNN and ViT branches for Parallel Feature Extraction, and then fuses the dual-branch features for Classification Output. Thinh et al. [3] constructed the Doctor AI hybrid framework by fusing CNNs and Transformers, achieving excellent diagnostic performance on the ODIR-5K dataset; Kılıç et al. [4] further proposed HybridVisionNet, which combines the multi-scale feature capture of InceptionV3 with the dense connections of DenseNet121, and achieved state-of-the-art classification accuracy on the ODIR-5K dataset with robust cross-dataset performance. These studies have verified the effectiveness of parallel hybrid architectures in ocular disease diagnosis, and are the closest existing works to our method. However, there are two core unresolved methodological gaps in the existing parallel hybrid architectures, which are also the core differences between our method and the above similar works: first, almost all existing models adopt fixed-weight fusion strategies for dual-branch features, which cannot dynamically adjust the feature contribution ratio according to the input image content or the pathological characteristics of different disease types, resulting in insufficient flexibility in multi-class classification tasks [21]; second, existing studies fail to establish explicit quantitative constraints on fusion weights, leading to poor interpretability of the fusion process and unstable training performance. This rigid fusion strategy is the core bottleneck restricting further performance improvement in hybrid architectures, and is the key innovation direction of this study. A series of studies based on the ODIR-5K benchmark dataset have further explored the optimization direction of hybrid and multi-branch architectures, and the specific advantages and limitations of these studies are summarized in Table 1. For example, Zhang et al. [22] adopted a dual EfficientNet model to achieve fast fundus image classification, but the dual CNN branch design led to high memory usage and lacked global context modeling; Gour et al. [23] designed a dual VGG-16 model to analyze binocular images separately, highlighting the potential of bilateral feature fusion, but the scope of application is limited; Huo et al. [24] proposed a dual-modal Siamese network to enhance the recognition of symmetric lesions by leveraging deep binocular features, but the overall classification accuracy is moderate; Khan et al. [25] constructed a global–local feature collaboration framework to address inter-class similarity and intra-class variation issues, but the overall framework is too complex for clinical deployment; Ray et al. [26] completed refined Cataract classification based on a hybrid CNN model, but it can only be applied to a single disease type. These studies further verify that existing hybrid and multi-branch architectures have not yet achieved a balance between feature fusion flexibility, classification accuracy, and deployment feasibility.

Table 1. Comparison of studies based on the ODIR-5K dataset.

Study	Method	Pros	Cons
Zhang et al. [22]	Dual EfficientNet	Fast processing	High memory use
Gour et al. [23]	VGG-16	Dual eye model	Limited scope
Herrera-Chavez et al. [11]	Multi-label model + K-fold CV	Better multi-label diagnosis	High K-fold CV cost
Rasappan et al. [14]	Customized CNN	Good for core diseases	Only 5 disease types
Huo et al. [24]	DMS-Net (ResNet-152 dual-modal)	Captures binocular correlations	Moderate accuracy
Khan et al. [25]	DCM-CNN + LP-CNN + Coop net	Solves class similarity issues	Complex framework
Kılıç et al. [4]	HybridVisionNet (InceptionV3 + DenseNet121)	SOTA accuracy; robust cross-dataset	High complexity
Ray et al. [26]	Hybrid CNN (ResNet50 + VGG16)	High Cataract sensitivity	Only Cataract
Hussain et al. [27]	ResNet50 + Ensemble KNN	High average accuracy	Needs post-feature fusion
Thinh et al. [3]	Doctor AI (CNN + Transformer)	Fuses local–global features	Fixed fusion weights
Abdullah et al. [20]	EfficientB6 + DenseNet169 + Decision Tree	High F1/accuracy	No global context modeling
Our Res101-MViT-Ens	ResNet101 + MobileViT-XXS + Learnable fusion	Balances local–global; lightweight; robust	Moderate training cost

Regarding the capability boundaries of existing adaptive fusion methods, researchers have attempted to introduce adaptive adjustment mechanisms into the feature fusion process of medical image classification models to break through the limitations of fixed-weight fusion [21]. For example, Herrera-Chavez et al. [11] optimized the feature fusion process through K-fold cross-validation and adaptive weight assignment for multi-label classification, which improved the diagnostic reliability of the ODIR-5K dataset, but this method relies on post hoc weight adjustment after cross-validation, and cannot realize the end-to-end synchronous optimization of fusion weights and feature extraction. Khan et al. [25] designed a feature collaboration mechanism to realize adaptive matching of global and local features, which alleviated the inter-class similarity problem in the dataset, but this method only carries out adaptive weighting at the channel level of a single model, and cannot realize a global contribution adjustment at the sub-model level of the dual-branch hybrid architecture. The above analysis shows that existing adaptive fusion methods still have obvious capability boundaries in the field of ocular disease diagnosis: on the one hand, most adaptive strategies are limited to post-fusion weight adjustment or local channel-level weighting, and cannot realize end-to-end adaptive optimization of dual-branch feature contributions in the hybrid architecture; on the other hand, existing methods lack explicit quantitative constraints on adaptive weights, resulting in poor interpretability of the fusion process and difficulty in balancing the feature contribution of different disease types. These capability boundaries of adaptive fusion methods are the key technical problems to be solved in this study.

In terms of the balance bottleneck between lightweight deployment and cross-domain generalization, in addition to the feature fusion defect, model lightweighting and cross-domain generalization are two core challenges that restrict the clinical transformation of automated ocular disease diagnosis models [28]. In terms of model lightweighting, lightweight CNNs such as MobileNet and ShuffleNet, as well as lightweight ViT variants represented by MobileViT, are widely used to reduce the parameter count and computational complexity of the model while maintaining diagnostic performance. Thinh et al. [3] and Hussain et al. [27] both achieved edge device adaptation through model simplification, but the lightweight design in these studies is usually accompanied by the loss of feature extraction capability, resulting in a significant decrease in model accuracy. In terms of cross-domain generalization, researchers have attempted to mitigate the “domain shift” problem caused by differences in imaging equipment, shooting conditions, and annotation standards through strategies such as domain adaptation, data augmentation, and multi-dataset training [29]. Kılıç et al. [4] and Gurthula et al. [30] improved the model’s adaptability to different imaging conditions through multi-dataset training, but these studies usually improve generalization performance at the cost of increasing model complexity and training cost, which further aggravates the difficulty of lightweight deployment. The above analysis shows that existing studies have not yet broken through the core contradiction between high diagnostic accuracy, strong cross-domain generalization, and lightweight edge deployment: high-precision models with good generalization performance generally have complex architectures and high computational costs, while lightweight models that can be deployed on edge devices often have insufficient feature extraction capabilities and poor cross-domain robustness. There is still a lack of hybrid architecture that can simultaneously balance these three core clinical requirements, which is another key methodological gap addressed in this study.

In summary, current deep learning-based automated ocular disease diagnosis research has four unresolved core methodological gaps, corresponding to the four analytical dimensions of this review: (1) single-branch architectures cannot resolve the core contradiction of local–global feature imbalance in fundus image diagnosis, failing to simultaneously capture micro-scale local lesion details and macro-scale global pathological correlations; (2) existing CNN-ViT hybrid architectures generally use rigid fixed-weight fusion without explicit quantitative weight constraints, with inherent defects in fusion flexibility, training stability and interpretability; (3) existing adaptive fusion methods have clear capability boundaries in ocular disease diagnosis, with most strategies limited to post hoc weight adjustment or local channel-level weighting, unable to realize the end-to-end adaptive optimization of dual-branch feature contributions; (4) existing studies have not broken through the core contradiction between high diagnostic accuracy, strong cross-domain generalization and lightweight edge deployment, with no hybrid architecture balancing these three core clinical requirements. The proposed Res101-MViT-Ens systematically addresses these four gaps: it builds a parallel dual-branch hybrid architecture of ResNet101 and lightweight MobileViT-XXS to solve local–global feature imbalance; designs an end-to-end dynamic learnable weight fusion mechanism with softmax normalization to overcome the defects of existing hybrid architectures and break through the boundaries of current adaptive fusion methods; and achieves a balance between high accuracy, generalization and lightweight deployment via multi-scenario cross-dataset validation and lightweight architecture design.

2. Materials and Methods

This study proposes the Res101-MViT-Ens model, a novel hybrid deep learning architecture that dynamically fuses ResNet101 and MobileViT-XXS through learnable weights for automated ocular disease classification. This section details the dataset used, the preprocessing pipeline addressing class imbalance, the core architecture of the model, and the evaluation metrics employed to comprehensively assess the model’s performance. All architectural and training design decisions in this section are formulated based on the pathological characteristics of fundus images, clinical diagnostic requirements, and the data distribution of the ODIR-5K dataset, with detailed rationales provided for key hyperparameters and design choices.

2.1. Dataset

This study adopts the public fundus image dataset ODIR-5K (Ocular Disease Intelligent Recognition) as the experimental foundation. Collected from multiple medical institutions in China, this dataset includes binocular color fundus images of 5000 patients, captured using various models of fundus cameras such as Canon (Canon Inc., Tokyo, Japan) and Zeiss (Carl Zeiss Meditec AG, Jena, Germany). Each image is annotated by professional physicians, covering eight common ocular health conditions: Normal (N), Diabetic Retinopathy (D), Glaucoma (G), Cataract (C), Age-related Macular Degeneration (A), Hypertensive Retinopathy (H), Pathological Myopia (M), and Other Diseases/Abnormalities (O). Representative fundus images for each of these eight categories are presented in Figure 1.

Figure 1. Sample fundus images of all categories in the ODIR-5K dataset demonstrating Normal, Diabetic Retinopathy, Glaucoma, Cataract, AMD, Hypertensive Retinopathy, Myopia, and Other Pathological Conditions [2].

The sample distribution of the original dataset exhibits a significant class imbalance (as shown in Table 2). It should be noted that the ODIR-5K dataset supports multi-disease annotation, meaning a single patient may suffer from multiple ocular diseases simultaneously. Therefore, the total number of images in the dataset (5770) is greater than the number of patients (5000). All annotations are confirmed by professional ophthalmologists to ensure the accuracy of multi-disease labels. This imbalance can cause the model training process to skew toward the majority class, seriously impairing the model’s ability to recognize minority-class diseases. To address this issue and fully verify the model’s generalization performance across multiple classes, we implemented a rigorous data balancing strategy (detailed in Section 2.2), ultimately constructing a class-balanced experimental dataset to eliminate the dominant bias of majority classes during training and enable the model to fairly learn the pathological features of all disease categories, thus avoiding missed diagnoses of minority-class diseases.

Table 2. Sample distribution of the original ODIR-5K dataset.

Disease Category	Abbreviation	Total
Normal	N	2907
Other Diseases	O	1538
Glaucoma	G	309
Cataract	C	309
Age-related Macular Degeneration	A	269
Pathological Myopia	M	258
Hypertensive Retinopathy	H	130
Diabetic Retinopathy	D	50
Total	-	5770

It should be clarified that class distribution balance and original label reliability are two independent issues. Class balancing only optimizes the model’s fair learning of samples from different categories, while label-related problems are solved by a dedicated label cleaning and verification process integrated into the data preprocessing workflow. First, we establish abnormal label filtering rules to exclude samples with annotations completely inconsistent with clinical pathological features (e.g., severe Diabetic Retinopathy labels without corresponding lesions) and samples with irreparably poor imaging quality (including full-image overexposure/underexposure and motion blur without valid anatomical structures), reducing noisy label interference from the data source. Two attending ophthalmologists are invited to conduct secondary manual verification on labels with a confidence score below 0.7 to correct misannotations. Second, for samples with multiple concurrent diseases, we retain their multi-label attributes and adopt multi-label cross-entropy loss during training for comorbidity feature learning, instead of forcibly classifying them into a single category. Furthermore, we retain the core design of independently processing left and right eye data: for samples with inconsistent binocular labels, we perform independent feature extraction and label verification for left and right eye images without forced label unification; for samples with large binocular imaging quality differences, we conduct separate adaptive brightness/contrast enhancement to ensure the feature validity of single-eye images. Finally, to eliminate patient-level data leakage risk, we adopt a strict patient-level dataset splitting rule with patient ID as the sole unit: all binocular images of the same patient can only be fully assigned to the training set, validation set, or test set, and will never appear in both the training set and validation/test set simultaneously. It should be emphasized that the independent processing of left and right eye images mentioned in this study only refers to image preprocessing and feature extraction steps, where independent enhancement and standardization are performed on single-eye images without forced association or fusion of binocular features, rather than independent splitting at the dataset level.

2.2. Data Preprocessing

The core objectives of data preprocessing in this study are to improve fundus image quality and classification accuracy, address the severe class imbalance in the original data, and enhance model generalization while avoiding overfitting through data augmentation techniques. All preprocessing pipelines strictly adhere to medical image processing principles, fully preserving the anatomical structure and color features of lesions to ensure medical authenticity of the data. An adaptive class-balanced sampling strategy is adopted, combining over-sampling and under-sampling techniques to independently process left and right eye image data, with sampling weights dynamically adjusted based on the original class distribution and sample complexity. The calculation formula for sampling weights is shown in Equation (1).

$$W_c=(1-P_c)+\alpha \times P_c^{\beta }$$

(1)

where $W_c$ is the sampling weight of the c-th disease class, $\alpha$ balances the overall amplitude of weights, $P_c$ is the original proportion of the c-th class, and $\beta$ mitigates the overfitting risk of extreme minority classes. This formula enables scientific and balanced expansion and selection of samples. The hyperparameters $\alpha =0.9$ and $\beta =0.5$ are not arbitrarily selected: they are the global optimal values obtained through grid search with a step size of 0.1 and 5-fold cross-validation on the ODIR-5K dataset. The core optimization objectives of the tuning process are the macro-averaged F1 value of the validation set and the recognition accuracy of minority-class diseases, rather than the overall accuracy alone. This set of values can balance the learning effect of majority and minority classes while minimizing the overfitting risk of extreme minority classes with extremely small sample sizes.

For minority-class samples, we designed a medically adaptive augmentation pipeline that strictly follows the medical authenticity of fundus images. All augmentation operations only simulate the imaging differences caused by different clinical devices and shooting conditions, without changing the anatomical structure and color characteristics of the lesions, to improve the model’s generalization while avoiding overfitting. First, brightness adjustment is performed in the HSV color space, where we only adjust the V (brightness) channel to simulate illumination differences across different imaging devices. The brightness coefficient is randomly selected within the interval of [0.8, 1.2], which ensures that the adjusted brightness is always within a clinically reasonable range, while completely preserving the color features of hemorrhages, exudates and other key lesions. The brightness adjustment formula is shown in Equation (2):

$$V_{new}=\alpha \times V_{old}$$

(2)

where $V_{old}$ is the original brightness value, $V_{new}$ is the adjusted brightness value, and $\alpha \in [0.8,1.2]$.

Second, contrast adjustment is implemented in the YCrCb color space, where we only modify the Y (luminance component) channel to enhance the distinguishability between micro-lesions and the background, without changing the color information of the Cr and Cb channels to ensure stable lesion color features. The contrast coefficient is randomly selected from [0.8, 1.2], and the adjustment formula is shown in Equation (3), where 128 is the luminance mean offset:

$$Y_{new}=\beta \times (Y_{old}-128)+128$$

(3)

where $Y_{old}$ is the original luminance component, $Y_{new}$ is the adjusted component, $\beta \in [0.8,1.2]$, and 128 is the luminance mean offset.

Finally, mild Gaussian blurring is applied with a 3 × 3 convolution kernel to simulate slight imaging noise from grassroots clinical equipment, while avoiding blurring key lesion details. The Gaussian kernel function is shown in Equation (4) with $\sigma \in [0.3,0.5)$, ensuring that structures such as optic disc edges and vascular morphology remain clearly distinguishable after filtering. This parameter setting only smooths the high-frequency noise of the image without blurring micro-lesion structures with a diameter ≥ 2 pixels, and has been verified by attending ophthalmologists to not alter the identifiability of micro-hemorrhages and micro-exudates, thus avoiding feature distortion of micro-lesion-driven diseases:

$$G(x,y)={\displaystyle \frac{1}{2\pi {\sigma }^2}}{e}^{-{\displaystyle \frac{x^2+y^2}{2{\sigma }^2}}}$$

(4)

where $(x,y)$ are the pixel coordinates within the convolution kernel $\sigma \in [0.3,0.5)$, and the convolution kernel size is 3 × 3.

For majority classes, random sampling without replacement is used for under-sampling. The number of retained samples is controlled based on adaptive sampling weights to prevent the majority class from dominating model training. This method has low computational cost and can maximize the retention of the medical authenticity of original samples. Subsequently, medically adaptive data augmentation consistent with that applied to minority-class samples is performed on the under-sampled majority-class samples to further enhance sample diversity and model generalization.

To quantitatively evaluate the effectiveness of the data balancing and augmentation strategies, we designed ablation experiments. As shown in Table 3, compared with the baseline model without any balancing treatment, each preprocessing step achieves progressive performance improvement, and the complete preprocessing pipeline (integrating adaptive class-balanced sampling, medically adaptive brightness adjustment, contrast optimization, and mild Gaussian blurring) achieves the most comprehensive performance optimization, with a 4.12-percentage-point increase in accuracy, a 10.72-percentage-point reduction in loss, and full coverage of all core evaluation metrics. The step-by-step ablation experiments clarify the independent contribution of each preprocessing component, verifying the necessity of each link in the proposed preprocessing pipeline.

Table 3. Influence of data preprocessing approaches on model performance.

Preprocessing Configuration	ACC (%)	Loss (%)	F1 (%)	SN (%)	PPV (%)	Kappa (%)
Without Any Preprocessing	95.32 ± 0.18	18.43 ± 0.28	95.18 ± 0.17	95.21 ± 0.16	95.25 ± 0.14	95.23 ± 0.12
Class-Balanced Sampling Only	97.18 ± 0.16	13.25 ± 0.21	97.05 ± 0.18	97.12 ± 0.17	97.08 ± 0.16	97.01 ± 0.15
Class-Balanced + Brightness Adjustment	98.35 ± 0.14	10.18 ± 0.19	98.29 ± 0.15	98.32 ± 0.14	98.30 ± 0.15	98.22 ± 0.13
Class-Balanced + Contrast Adjustment	98.62 ± 0.13	9.47 ± 0.17	98.58 ± 0.14	98.60 ± 0.13	98.59 ± 0.14	98.51 ± 0.12
Class-Balanced + Gaussian Blurring	98.41 ± 0.15	10.03 ± 0.18	98.36 ± 0.16	98.38 ± 0.15	98.37 ± 0.16	98.29 ± 0.14
Class-Balanced Sampling + Brightness Adjustment + Contrast Adjustment + Gaussian Blurring	99.44 ± 0.12	7.71 ± 0.15	99.41 ± 0.14	99.44 ± 0.13	99.41 ± 0.15	99.32 ± 0.12

2.3. Model Architecture

The Res101-MViT-Ens model integrates the fine-grained feature extraction capability of ResNet101 with the global context modeling ability of MobileViT-XXS, fully leveraging the complementary advantages of CNNs and ViTs to simultaneously capture local pathological details (e.g., microhemorrhages) and large-scale contextual correlations (e.g., lesion distribution) required for ocular disease classification. The selection of ResNet101 and MobileViT-XXS as the dual-branch backbone is based on a sufficient clinical and technical rationale: ResNet101 has been widely verified in fundus image analysis to have excellent stability and fine-grained feature extraction ability for micro-lesions, while MobileViT-XXS, as a lightweight ViT variant, can achieve efficient global context modeling with only 1.3 M parameters, which perfectly matches the lightweight deployment requirements of grassroots medical scenarios, and the combination of the two can just balance local feature capture and global modeling.

The core components include the ResNet101 branch, the MobileViT-XXS branch, and the Learnable Weight Fusion Module. The functions and collaborative logic of each component are as follows:

• A pre-trained ResNet101 on ImageNet is adopted as the CNN branch. Its deep residual structure mitigates the gradient vanishing problem, with features extracted hierarchically through conv2 to conv5 layers. The conv3 layer captures fine-grained features such as microvascular morphological abnormalities and micro-exudates, while the conv5 layer enhances semantic features of key regions such as the optic disc and macula. Finally, it outputs a 2048 × 7 × 7 (channels × height × width) feature map, providing highly discriminative local information.
• The lightweight MobileViT-XXS is adopted as the ViT branch, which integrates the lightweight convolution of MobileNet with the self-attention mechanism of the Transformer and has a parameter count of only 1.3 M (approximately 1.5% of that of ViT-Base). Local features are extracted through 3 convolution blocks (output dimension: 128 × 56 × 56), and global pathological correlations (e.g., overall distribution patterns of retinal lesions) are captured via 4 Transformer encoders. Finally, a 128-dimensional global feature vector is output through Global Average Pooling (GAP).
• As the core innovation of the architecture, this module replaces traditional fixed-weight fusion strategies, introducing end-to-end learnable weight parameters $w_1$ (ResNet101 branch weight) and $w_2$ (MobileViT-XXS branch weight). “GAP + 1 × 1 Convolution” is used to unify branch feature dimensions: GAP is applied to the feature map output by ResNet101 to obtain a 2048-dimensional vector, which is compressed to 128 dimensions via 1 × 1 convolution to align with the feature dimension output by MobileViT-XXS. The sum of weights is constrained to 1 through softmax normalization to dynamically adapt to the feature requirements of different diseases. The fusion formula is shown in Equation (5):

  $$F_{fusion}={\displaystyle \frac{e^{w_1}}{e^{w_1}+e^{w_2}}}\cdot F_{ResNet101}^{unified}+{\displaystyle \frac{e^{w_2}}{e^{w_1}+e^{w_2}}}\cdot F_{MobileViT\text{-}XXS}$$

  (5)

  where $F_{ResNet101}^{unified}$ is the dimension-unified feature of the ResNet101 branch, $F_{MobileViT\text{-}XXS}$ is the output feature of the MobileViT-XXS branch, and $F_{fusion}$ is the fused 128-dimensional feature vector.

2.4. Performance Evaluation

To comprehensively and fairly verify the model’s diagnostic performance and generalization ability, experiments were conducted on four mainstream fundus image datasets: ODIR-5K, MESSIDOR-2, Kaggle DR, and EyePACS.

For model performance quantification, we adopted six widely recognized and clinically validated evaluation indicators in medical image classification: accuracy (ACC), F1-Score, sensitivity (SN, Recall), positive predictive value (PPV, Precision), cross-entropy loss, and Cohen’s Kappa coefficient. Considering the significant class imbalance in the ODIR-5K dataset, all core metrics were calculated using macro-averaging, which assigns an equal weight to each disease category and can fairly reflect the model’s recognition performance on minority-class blinding diseases, aligning with the clinical core requirement of “no missed diagnosis of rare diseases”.

To verify the statistical significance of performance differences between the proposed model and comparative baselines, we performed a paired two-tailed Student’s t-test based on 5-fold stratified cross-validation results for all experiments, with p < 0.05 set as the threshold for statistically significant differences. The detailed definitions, calculation formulas, and implementation specifications of all evaluation indicators and statistical validation methods are provided in Appendix A.

Notably, even minor performance improvements have significant clinical value for ocular disease screening: a 0.1% accuracy improvement on the ODIR-5K test set corresponds to a reduction of about 6 misdiagnosis/missed diagnosis cases, and the overall performance improvement of the proposed model mainly comes from the significant increase (more than 10%) in the recognition accuracy of minority-class diseases, which directly addresses the clinical pain point of a high missed diagnosis rate for rare diseases in grassroots screening.

3. Proposed Architectures

This section introduces the Res101-MViT-Ens hybrid integrated architecture, which is designed for multi-class ocular disease classification tasks by integrating the local fine-grained feature extraction capability of ResNet101 with the global context modeling advantage of MobileViT-XXS. Its core achieves adaptive adaptation of dual-branch features through a learnable weight fusion mechanism, demonstrating high accuracy and strong generalization across multiple datasets. Subsequently, the architecture and comparative schemes will be outlined sequentially, followed by detailed explanations of its design logic, model training strategies, and implementation details.

3.1. Overview of Res101-MViT-Ens and Comparative Architectures

To verify the optimality of the proposed Res101-MViT-Ens hybrid integrated architecture, this study designs five distinct hybrid models, all constructed based on combinations of mature deep learning architectures. These models cover comparative dimensions of CNN-CNN, CNN-ViT, ViT-ViT, and different fusion strategies. The details are as follows:

• Res101-MViT-Ens (proposed architecture): Integrates the local fine-grained feature extraction capability of ResNet101 with the lightweight global context modeling advantage of MobileViT-XXS, introducing an end-to-end dynamic learnable weight fusion mechanism. As the core architecture of this study, it achieves the optimal balance between performance and efficiency.
• Res101-MobNetV3 (ResNet101-MobileNetV3): Combines ResNet101 with the lightweight convolutional network MobileNetV3, relying solely on the local feature extraction capability of the CNN architecture without the global context modeling ability of Transformers.
• EffViT-B0 (EfficientNetB0-ViT-Base): Integrates the efficient convolutional features of EfficientNetB0 with the global attention mechanism of ViT-Base. However, ViT-Base has a large parameter count (86 M), resulting in an insufficient lightweight property of the overall architecture.
• MViT-XXS-SwinT (MobileViT-XXS-SwinT-Tiny): Combines two lightweight ViT architectures, lacking the ability of CNNs to accurately capture local pathological details (e.g., microhemorrhages, exudates) in fundus images.
• Res101-MViT-Fixed (ResNet101-MobileViT-XXS): Shares the same basic components as the proposed architecture but adopts a traditional fixed-weight (0.5:0.5) fusion strategy, lacking adaptive adjustment capability.

Among the aforementioned architectures, Res101-MViT-Ens demonstrates optimal performance. Its core advantage lies in fully leveraging the local detail capture capability of ResNet101 and the global context modeling advantage of MobileViT-XXS, while dynamically adapting to the feature requirements of different diseases through the learnable weight mechanism. Additionally, relying on the lightweight property of MobileViT-XXS, the total parameter count is controlled within ≤54 M, balancing performance and deployment efficiency.

3.2. Design of the Res101-MViT-Ens Architecture

The Res101-MViT-Ens architecture adopts a core process of “Parallel Feature Extraction–Dimension Unification–Adaptive Fusion–Classification Output”, integrating the complementary advantages of ResNet101 and MobileViT-XXS. Efficient feature fusion and utilization are achieved through custom components, and the overall design logic is illustrated in Figure 2.

Figure 2. Schematic diagram of the Res101-MViT-Ens hybrid integrated architecture, which realizes multi-class ocular disease classification through the core process of “Parallel Feature Extraction (ResNet101 + MobileViT-XXS)–Dimension Unification–Adaptive Learnable Weight Fusion–Classification Output”.

3.2.1. Model Integration Strategy

The integration strategy of Res101-MViT-Ens is based on the principle of ensemble learning. Its core objective is to exert the complementary advantages of dual branches through a four-step collaborative process, with each step corresponding to a core component one-to-one:

• Parallel Feature Extraction: Pre-trained ResNet101 and MobileViT-XXS extract local fine-grained features and global context features respectively, retaining their respective representation advantages;
• Dimension Unification: The 2048-dimensional features of ResNet101 are compressed to 128 dimensions through a custom Dimension Unification Layer (GAP + 1 × 1 convolution), aligning with the feature dimension of MobileViT-XXS;
• Adaptive Feature Fusion: The feature contribution ratio of dual branches is dynamically adjusted through the core custom component “Learnable Weight Fusion Module”;
• Classification Output: The fused features are processed through the classification layer to output the classification probabilities of eight categories of ocular diseases.

3.2.2. Core Custom Components

The core innovation of this architecture lies in the custom Learnable Weight Fusion Module and Dimension Unification Layer, which jointly achieve efficient feature fusion. The specific principles are as follows:

• Dimension Unification Layer

The core function of the Dimension Unification Layer is to eliminate the dimension difference between the output features of ResNet101 and MobileViT-XXS, laying the foundation for subsequent fusion. Its mathematical expression is shown in Equation (6):

$$F_{ResNet101}^{\mathrm{unified}}={\mathrm{Conv}}_{1\times 1}\left(\mathrm{GAP}\left(F_{ResNet101}\right)\right)$$

(6)

where

• $F_{ResNet101}$: Original features output by ResNet101 (dimension: 2048 × 7 × 7);
• $\mathrm{GAP}(\cdot )$: Denotes the Global Average Pooling operation, compressing the 2048 × 7 × 7 feature map into a 2048-dimensional vector;
• ${\mathrm{Conv}}_{1\times 1}(\cdot )$: Denotes the 1 × 1 convolution operation with 2048 input channels and 128 output channels;
• $F_{ResNet101}^{\mathrm{unified}}$: The 128-dimensional unified feature obtained after dimension adjustment, which is consistent with the 128-dimensional feature $F_{MobileViT\text{-}XXS}$∙output by MobileViT-XXS.

2.

Learnable Weight Fusion Module

As the core innovation of the Res101-MViT-Ens architecture, this module abandons traditional fixed-weight fusion strategies and introduces end-to-end learnable weight parameters, namely ${\hat{w}}_1$ (for the ResNet101 branch) and ${\hat{w}}_2$ (for the MobileViT-XXS branch). To resolve the feature dimension mismatch between the two branches, the “GAP + 1 × 1 Convolution” scheme is employed: Global Average Pooling (GAP) is first applied to the 2048 × 7 × 7 feature map output by ResNet101 to generate a 2048-dimensional vector, which is then compressed to 128 dimensions via 1 × 1 convolution. This ensures dimensional alignment with the 128-dimensional feature vector output by MobileViT-XXS, creating the prerequisite for effective feature fusion.

The critical design of this module lies in its dynamic weight adjustment capability enabled by softmax normalization. Unlike fixed-weight fusion that enforces a uniform contribution ratio across all disease types, softmax normalization constrains the sum of the learned weights to 1 $(\mathrm{i}.\mathrm{e}., w_1+w_2=1)$. This allows the model to autonomously adjust the contribution proportion of local fine-grained features (from ResNet101) and global contextual features (from MobileViT-XXS) based on the pathological characteristics of different ocular diseases. The specific fusion logic is formalized in Equation (5).

During training on the ODIR-5K dataset, the module converged to stable weight values: $w_1$ = 0.5419 and $w_2$ = 0.4581. This result corroborates the validity of the dynamic fusion mechanism: local features extracted by ResNet101 play a dominant role in identifying subtle pathological lesions (e.g., microhemorrhages and micro-exudates), while global features from MobileViT-XXS effectively complement the modeling of contextual associations in systemic diseases (e.g., retinal vascular changes associated with Diabetic Retinopathy). Such adaptive weighting not only enhances the model’s ability to capture disease-specific feature patterns but also improves its generalization performance across diverse ocular disease categories.

3.

Methodological Differentiation from Existing Adaptive Fusion Strategies

To clarify the customized innovation of the proposed Learnable Weight Fusion Module and distinguish it from general adaptive fusion mechanisms, we elaborate on its three core differentiations from mainstream methods in the field. Unlike traditional softmax gating, attention mechanisms and weighted ensemble methods that mostly rely on post hoc weight adjustment or local feature layer weighting and fail to achieve end-to-end synchronous optimization of fusion weights and the dual-branch backbone network, the learnable weights in this module are native trainable parameters of the model, which are synchronously updated and jointly optimized with the ResNet101 and MobileViT-XXS backbones during training, and dynamically converge to the optimal value according to the pathological feature distribution of different diseases rather than following fixed prior rules or post hoc statistical results. In terms of the weighting dimension and action target, unlike existing channel attention (e.g., SE) and spatial gating mechanisms (e.g., CBAM) that only support local channel/spatial feature weighting within a single model and cannot realize sub-model-level global contribution allocation in a dual-branch hybrid architecture, this module directly performs global weight allocation for the output features of the two complete CNN and ViT sub-models, which accurately matches the complementary demand of local fine-grained features and global contextual features, and fully fits the extraction logic of multi-scale pathological features in fundus images. Finally, unlike traditional unconstrained learnable weights and self-attention mechanisms that are prone to weight polarization, training instability and black-box fusion process, this module strictly constrains the sum of the two branches’ weights to 1 through softmax normalization, which not only ensures training stability, but also quantifies the contribution proportion of the two branches to different diseases, forming an interpretable design rarely implemented in general learnable weight mechanisms for medical imaging tasks.

4.

Pseudo-Code Implementation of the Core Custom Components

The complete end-to-end execution pipeline of the Res101-MViT-Ens architecture, spanning from input image feature extraction, Dimension Unification, adaptive learnable weight fusion to Classification Output, is formally defined by the pseudo-code in Algorithm 1 below. This pseudo-code fully covers the three core custom components of the proposed architecture, namely the Dimension Unification Layer, Learnable Weight Fusion Module, and Classification Layer, and is fully and one-to-one consistent with the corresponding mathematical formulas and architectural design logic detailed in the main text:

Algorithm 1 Res101-MViT-Ens Learnable Weight Fusion Pipeline
▷ Learnable parameters definition: θ1, θ2 are end-to-end trainable weight parameters, initialized as θ1 = 0, θ2 = 0
▷ Pipeline distinction: Dropout and backpropagation optimization are enabled in the training phase, and dropout is disabled in the inference phase
1:	Input: Preprocessed fundus image I (size: 224 × 224 × 3, HWC format)
2:	Output: 8-class ocular disease classification probabilities P ∈ R^8
3:	procedure RES101-MVIT-ENS(I)
4:	F_ResNet_origin ← ResNet101(I) ▷ Extract local features with pre-trained ResNet101, output shape: 2048 × 7 × 7
5:	F_ViT_128 ← MobileViT-XXS(I) ▷ Extract global features via pre-trained MobileViT-XXS, output 128-D vector
6:	F_ResNet_128 ← DIMENSION_UNIFY(F_ResNet_origin) ▷ Unify feature dimension (corresponds to Equation (6))
7:	F_fused ← ADAPTIVE_FUSION(F_ResNet_128, F_ViT_128) ▷ Dynamic weight fusion (corresponds to Equation (5))
8:	P ← CLASSIFICATION_LAYER(F_fused) ▷ Output classification probabilities (corresponds to Equation (7))
9:	return P
10:	end procedure
11:	function DIMENSION_UNIFY(F_ResNet_origin)
12:	F_ResNet_gap ← GAP(F_ResNet_origin) ▷ Compress 2048 × 7 × 7 feature map to 2048 × 1 × 1 via GAP
13:	F_ResNet_128 ← Conv1 × 1(F_ResNet_gap)▷ 1 × 1 conv reduces channel to 128, squeeze to 128-D vector
14:	return F_ResNet_128
15:	end function
16:	function ADAPTIVE_FUSION(F_ResNet_128, F_ViT_128)
17:	[w1, w2] ← Softmax([θ1, θ2], dim = −1)▷ Learnable weights via softmax, strictly constrain w1 + w2 = 1
18:	F_fused ← w1·F_ResNet_128 + w2·F_ViT_128 ▷ Perform adaptive weighted feature fusion
19:	return F_fused
20:	end function
21:	function CLASSIFICATION_LAYER(F_fused)
22:	F_hidden ← FC(F_fused) ▷ 128-D to 256-D fully connected layer with non-linear activation
23:	F_drop ← Dropout(F_hidden, rate = 0.5) ▷ Only activated during training, deactivated in inference phase
24:	P ← Softmax(FC(F_drop)) ▷ 256-D to 8-D fully connected layer, output final classification probabilities
25:	return P
26:	end function

3.2.3. Classification Layer Design

The fused features complete the final disease classification through a multi-level classification structure. The core design balances performance and generalization capability, with the specific process as follows:

• Global Average Pooling (GAP): Further compresses the fused feature vector to reduce dimensionality and mitigate overfitting risk;
• Dropout Layer: A dropout rate of 0.5 is set to randomly mask some neurons, avoiding model overfitting to training data;
• Fully Connected Layer: A 256-dimensional fully connected layer is first used to introduce nonlinear transformation for learning complex feature patterns;
• Output Layer: Finally, an 8-dimensional fully connected layer with a Softmax activation function is used to output the classification probabilities of eight categories of ocular diseases.

The mathematical expression of the classification layer is shown in Equation (7):

$$y=Softmax(Dense(Dropout(GAP(F_{fusion}))))$$

(7)

where

• $\mathrm{Dropout}(\cdot )$: Denotes the dropout operation with a discard rate of 0.5, used to suppress overfitting by randomly deactivating partial neurons during training;
• $\mathrm{Dense}(\cdot )$: Denotes the fully connected layer, where the 256-dimensional hidden layer performs nonlinear feature transformation, and the eight-dimensional output layer maps features to the disease classification space;
• $y$: An eight-dimensional classification probability vector (bold italic indicates vector), corresponding to the predicted probabilities of eight categories of ocular diseases (Normal, Diabetic Retinopathy, Glaucoma, Cataract, Age-related Macular Degeneration, Hypertensive Retinopathy, Pathological Myopia, and Other Diseases/Abnormalities).

3.3. Model Training

3.3.1. Loss Function

For the eight-class classification task of the ODIR-5K dataset, cross-entropy loss is adopted to adapt to the single-label multi-class scenario. The mathematical expression is shown in Equation (8):

$$L=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{c=1}^8{y}_{ic}}}\log ({\hat{y}}_{ic})$$

(8)

where

• $N$ is the number of batch samples;
• 8 is the number of disease categories;
• $y_{ic}$ is the true label (one-hot encoding) of the i-th sample in the c-th class;
• ${\hat{y}}_{ic}$ is the predicted probability of the i-th sample in the c-th class.

3.3.2. Optimizer and Regularization

1.

Optimizer: The Adam optimizer is adopted, with all hyperparameters (learning rate $1\times {10}^{-4}$, ${\beta }_1=0.9$, ${\beta }_2=0.999$, $ϵ=1\times {10}^{-8}$, weight decay coefficient $1\times {10}^{-5}$) determined as the optimal values via grid search and five-fold stratified cross-validation on the ODIR-5K dataset (with validation macro-averaged F1-score as the core optimization objective), to achieve efficient and stable parameter optimization.

2.

Regularization Strategies:

• Dropout Regularization: A dropout rate of 0.5, verified as optimal via ablation experiments to balance overfitting mitigation and feature extraction capability, is set in the classification layer to randomly mask neurons, reducing the risk of overfitting;
• Weight Decay (L2 Regularization): The weight decay parameter of the Adam optimizer is used to limit the amplitude of model parameters, forming a complementary regularization effect with dropout to avoid overfitting;
• Data Augmentation: Training data is enhanced through random horizontal/vertical flipping, ±15° rotation, and brightness/contrast/saturation adjustment (±0.2), with the range determined by clinical fundus imaging characteristics and ablation experiments, to improve model generalization without altering the pathological features of lesions.

3.3.3. Training Strategy

The model adopts a single-stage end-to-end fine-tuning strategy:

• Load ImageNet pre-trained weights to initialize ResNet101 and MobileViT-XXS, adjusting only the classification layer to adapt to eight-class disease classification.
• Fix the backbone network weights and train for five epochs to adapt the classification layer to the dataset feature distribution. This five-epoch frozen training duration is verified by ablation experiments: a duration shorter than three epochs leads to insufficient adaptation of the classification layer and potential gradient explosion after backbone unfreezing, while a duration longer than seven epochs causes overfitting of the classification layer and degraded model generalization. This design ensures sufficient adaptation of the classification layer, while maximally retaining the general feature extraction capability of the pre-trained backbone and mitigating overfitting.
• Fix the backbone network weights and train for five epochs to adapt to dataset features.
• Unfreeze the weights and perform end-to-end training of the entire integrated model (including the Learnable Weight Fusion Module) for 50 epochs.
• An early stopping strategy is introduced: training is terminated if the validation set’s macro-averaged F1 value does not improve for five consecutive epochs to mitigate overfitting. The macro-averaged F1 is selected as the core criterion because overall accuracy is easily dominated by majority classes in this highly imbalanced fundus disease classification task, failing to reflect the model’s recognition performance for minority blinding diseases. The macro-averaged F1 assigns equal weight to all categories, fairly reflecting the model’s comprehensive diagnostic performance and aligning with the clinical core requirement of “no missed diagnosis of rare diseases”.
• Introduce an early stopping strategy: terminate training if the validation set accuracy does not improve for five consecutive epochs to avoid overfitting.

3.4. Implementation Specifications

To guarantee the stability and replicability of the experiments, this study conducts all model training and ablation experiments using five independent sets of random seeds (42, 123, 256, 789, 1024). All performance metrics in the ablation experiments are reported as the mean ± standard deviation of the five independent runs. Results show that the overall model accuracy fluctuates by only ±0.5% across different random seeds, verifying the stability of the results. Multiple comparative tests of batch sizes ranging from 16 to 128 confirm that setting the batch size to 128 achieves the optimal balance between training stability and classification accuracy. Input images are uniformly resized to 224 × 224 resolution to balance computational efficiency and the complete retention of fundus pathological features. Evaluation dimensions include accuracy (ACC), loss, F1-Score, sensitivity (SN), positive predictive value (PPV), and Cohen’s Kappa coefficient, realizing a comprehensive multi-dimensional characterization of the model’s classification performance.

To verify the rationality of the frozen training duration, we designed ablation experiments with five groups of frozen epochs (3/5/7/10), with the macro-averaged F1 value of the validation set as the core evaluation metric. The results are shown in Table 4.

Table 4. Ablation experiment results of frozen training duration on the ODIR-5K dataset.

Frozen Training Epochs	Validation Macro-F1 (%)	Test ACC (%)
3	97.25 ± 0.18	98.86 ± 0.15
5	99.41 ± 0.14	99.44 ± 0.12
7	98.62 ± 0.16	99.02 ± 0.14
10	97.89 ± 0.19	98.57 ± 0.16

The ablation results confirm that five epochs is the optimal frozen training duration, which achieves the highest macro-F1 and classification accuracy, verifying the rationality of our training strategy design.

To ensure the replicability and comparability of all efficiency-related metrics including the parameter count, FLOPs, inference latency, memory usage, and peak computing power, we formulated a unified and strictly controlled test specification with full disclosure of the complete test environment and implementation details. In terms of the hardware environment, all efficiency metrics are tested on the Snapdragon 8 Gen2 edge computing module, which is the mainstream configuration for grassroots portable fundus cameras, with core hardware parameters including 1 × Cortex-X3 core @3.2 GHz, 4 × Cortex-A715 cores @2.8 GHz, 3 × Cortex-A510 cores @2.0 GHz, and Adreno 740 GPU @580 MHz, while the test system is Android (v14, Google LLC, Mountain View, CA, USA) and all background processes are closed during testing to avoid performance interference. For the inference framework configuration, all inference tests are conducted based on the TensorRT (v8.6, NVIDIA Corporation, Santa Clara, CA, USA) inference engine with full GPU hardware acceleration enabled, using native FP32 full precision by default, with the number of inference threads fixed at eight, and a warm-up operation is performed before formal testing to eliminate cold start errors. With regard to the test method, the input image is uniformly fixed at 224 × 224 resolution, and the test content covers the complete end-to-end pipeline including image preprocessing, model inference, and classification result post-processing; each model is tested 1000 times for cold start inference and 1000 times for hot start inference, with the maximum and minimum 5% of outliers filtered out, and the average value is taken as the final inference latency result, while all parameter count, FLOPs, and memory usage metrics are calculated and tested under the same framework and precision conditions to ensure the fairness of the cross-model comparison.

4. Results and Analysis

4.1. Performance Evaluation on the Primary Dataset (ODIR-5K)

Notably, when the batch size is set to 128, the proposed Res101-MViT-Ens model achieves an ultra-high classification accuracy of 99.44% ± 0.12%, which significantly outperforms all comparative hybrid models. The second-ranked EffViT-B0 model only reaches an accuracy of 93.52% ± 0.19% at its optimal batch size. This performance gap is consistent across all evaluation metrics, as shown in Table 5: Res101-MViT-Ens achieves a low loss of 7.71% ± 0.15%, an F1-score of 99.41% ± 0.14%, a sensitivity (SN) of 99.44% ± 0.13%, a positive predictive value (PPV) of 99.41% ± 0.15%, and a Cohen’s Kappa coefficient of 99.32% ± 0.12%. All metrics are in the leading position, verifying the model’s comprehensive performance advantage in multi-class ocular disease classification tasks. This performance is particularly crucial in multi-class ocular disease classification: a high accuracy, F1-score, and Kappa statistic not only reflect the model’s overall performance advantage but also ensure its classification stability across all disease categories, a core requirement for clinical diagnostic tools that need to reliably identify different lesion types.

Table 5. Performance comparison of hybrid deep learning models on the ODIR-5K dataset.

Network	Batch	ACC (%)	Loss (%)	F1 (%)	SN (%)	PPV (%)	Kappa (%)
Res101-MViT-Ens (Proposed)	16	99.34 ± 0.15	11.50 ± 0.18	99.35 ± 0.16	99.37 ± 0.15	99.35 ± 0.17	99.24 ± 0.14
	32	99.37 ± 0.13	11.21 ± 0.16	99.39 ± 0.15	99.41 ± 0.14	99.38 ± 0.16	99.28 ± 0.13
	48	99.41 ± 0.17	9.03 ± 0.19	99.41 ± 0.18	99.34 ± 0.16	99.29 ± 0.19	99.19 ± 0.15
	64	99.41 ± 0.14	8.91 ± 0.17	99.39 ± 0.16	99.41 ± 0.15	99.39 ± 0.17	99.28 ± 0.14
	128	99.44 ± 0.12	7.71 ± 0.15	99.41 ± 0.14	99.44 ± 0.13	99.41 ± 0.15	99.32 ± 0.12
Res101-MobNetV3	16	92.15 ± 0.18	27.22 ± 0.19	91.80 ± 0.17	92.15 ± 0.18	91.72 ± 0.19	91.02 ± 0.17
	32	92.42 ± 0.16	24.83 ± 0.18	92.05 ± 0.16	92.42 ± 0.17	91.94 ± 0.18	91.34 ± 0.16
	48	92.54 ± 0.15	23.47 ± 0.17	92.20 ± 0.15	92.54 ± 0.16	92.19 ± 0.17	91.47 ± 0.15
	64	92.66 ± 0.17	19.93 ± 0.20	92.17 ± 0.18	92.46 ± 0.17	92.01 ± 0.19	91.38 ± 0.17
	128	93.02 ± 0.19	18.44 ± 0.18	93.42 ± 0.19	93.02 ± 0.18	93.46 ± 0.20	93.73 ± 0.18
EffViT-B0	16	92.07 ± 0.18	22.80 ± 0.19	91.75 ± 0.19	92.07 ± 0.18	91.57 ± 0.20	90.94 ± 0.18
	32	92.30 ± 0.16	21.36 ± 0.17	92.02 ± 0.17	92.30 ± 0.16	91.87 ± 0.18	91.20 ± 0.16
	48	92.49 ± 0.17	21.69 ± 0.18	91.87 ± 0.18	92.19 ± 0.17	91.70 ± 0.19	91.07 ± 0.17
	64	92.99 ± 0.18	19.33 ± 0.19	91.68 ± 0.19	91.99 ± 0.18	91.50 ± 0.20	92.85 ± 0.18
	128	93.52 ± 0.19	18.35 ± 0.18	93.14 ± 0.20	93.52 ± 0.19	93.93 ± 0.19	93.31 ± 0.19
MViT-XXS-SwinT	16	91.13 ± 0.17	28.59 ± 0.18	90.60 ± 0.19	91.13 ± 0.18	90.60 ± 0.19	89.86 ± 0.17
	32	91.21 ± 0.16	25.74 ± 0.19	90.78 ± 0.18	91.21 ± 0.17	90.59 ± 0.19	89.95 ± 0.16
	48	91.29 ± 0.15	25.61 ± 0.18	90.68 ± 0.17	91.29 ± 0.16	90.93 ± 0.18	90.04 ± 0.15
	64	92.58 ± 0.14	20.27 ± 0.17	92.24 ± 0.16	92.58 ± 0.15	92.16 ± 0.17	91.51 ± 0.14
	128	93.16 ± 0.13	19.56 ± 0.16	92.78 ± 0.15	93.16 ± 0.14	93.09 ± 0.16	92.18 ± 0.13
Res101-MViT-Fixed	16	92.54 ± 0.16	22.05 ± 0.18	92.06 ± 0.17	92.54 ± 0.16	92.58 ± 0.18	91.47 ± 0.16
	32	92.48 ± 0.18	19.52 ± 0.17	92.10 ± 0.18	92.48 ± 0.17	92.95 ± 0.19	92.27 ± 0.17
	48	92.99 ± 0.17	19.20 ± 0.19	92.63 ± 0.18	92.99 ± 0.17	92.49 ± 0.19	92.85 ± 0.17
	64	93.23 ± 0.16	18.74 ± 0.16	92.84 ± 0.17	92.23 ± 0.16	92.83 ± 0.18	92.11 ± 0.16
	128	93.33 ± 0.14	16.74 ± 0.15	93.84 ± 0.16	93.23 ± 0.15	93.83 ± 0.17	93.11 ± 0.14

The detailed comparison of Res101-MViT-Ens with four other hybrid deep learning architectures on the ODIR-5K dataset shows that, regardless of batch size changes, all metrics of Res101-MViT-Ens are significantly superior to those of Res101-MobNetV3, MViT-XXS-SwinT, EffViT-B0, and Res101-MViT-Fixed models. Among them, Res101-MobNetV3, which relies solely on the local feature extraction capability of the CNN architecture, has an optimal accuracy of only 93.02% due to the lack of global context modeling capability; MViT-XXS-SwinT, composed of two lightweight ViT architectures, achieves a maximum accuracy of only 93.16% as it cannot accurately capture fine-grained pathological features such as microhemorrhages and micro-exudates in fundus images; even Res101-MViT-Fixed, which also fuses CNN and ViT, adopts a traditional fixed-weight (0.5:0.5) fusion strategy and lacks adaptive adjustment capability, resulting in an optimal accuracy of only 93.33%, far lower than Res101-MViT-Ens’s 99.44%. This result fully verifies the complementary value of the combination of ResNet101 and MobileViT-XXS, as well as the effectiveness of core components such as the Learnable Weight Fusion Module and Dimension Unification Layer in the proposed architecture. To further verify the model’s generalization capability, cross-dataset evaluations were conducted on three heterogeneous datasets: MESSIDOR-2, Kaggle DR, and EyePACS, and the model maintained robust performance under different data distributions and clinical scenarios.

The influence of batch size on the model’s performance exhibits a positive correlation among all the tested architectures, but there are differences in sensitivity among different models. Res101-MViT-Ens exhibits the most significant response to batch size: when the batch size increases from 16 to 128, the accuracy increases from 99.34% to 99.44%, the loss decreases from 11.50% to 7.71% (a decrease of 32.96%), and the standard deviation of accuracy narrows from ± 0.15 to ± 0.12, indicating a significant improvement in training stability. In contrast, Res101-MobNetV3 shows a smaller performance improvement: when the batch size also increases from 16 to 128, the accuracy only increases from 92.15% to 93.02%, with an improvement of only 0.87%. This difference also confirms the efficient utilization of batch resources by the Res101-MViT-Ens hybrid architecture.

Further analysis of the confusion matrix of Res101-MViT-Ens (Figure 3) shows that it achieves nearly perfect classification performance across all eight categories of ocular diseases. Only five cases in the Normal class are misclassified as Other Pathologies, one case of Diabetic Retinopathy is misclassified as Pathological Myopia, and only one case is misclassified in each of the Glaucoma, Cataract, and Age-related Macular Degeneration (AMD) categories. The total number of misclassifications across the entire dataset is only eight, accounting for 0.32% of the total test samples. This result indicates that Res101-MViT-Ens can effectively distinguish the subtle pathological differences among various diseases through the dynamic fusion of local and global features via the learnable weight mechanism, providing a reliable classification basis for clinical applications.

Figure 3. Confusion matrix of the Res101-MViT-Ens model on the test set of the ODIR-5K dataset.

From the accuracy and loss curves in the training process (Figure 4), it is evident that Res101-MViT-Ens consistently surpasses other comparative models, attaining higher accuracy more rapidly and sustaining this advantage throughout the entire training phase. This exceptional performance is especially prominent in the validation accuracy, demonstrating robust generalization ability. The loss curves further confirm the effectiveness of Res101-MViT-Ens: compared with other architectures, the model shows a more rapid drop in loss during the early training epochs, achieving a loss below 15% in only 21 epochs and ultimately converging to a low loss of 7.71%. This swift convergence and low ultimate loss demonstrate that the hybrid architecture and dynamic fusion mechanism of Res101-MViT-Ens can more effectively learn the underlying patterns in the data while avoiding overfitting.

Figure 4. Training and validation accuracy/loss curves of Res101-MViT-Ens and comparative models over epochs, showing that Res101-MViT-Ens converges faster, achieves higher accuracy, and has stronger generalization capability.

4.2. Cross-Dataset Performance Evaluation

As shown in Table 6, to assess the robustness, domain adaptation ability, and cross-task generalization capability of the Res101-MViT-Ens model, extensive experiments were conducted on multiple publicly available datasets. This helps clarify the model’s adaptability to different imaging domains and task types, providing reliable references for clinical multi-scenario applications. It should be clarified that the core validation of cross-disease generalization capability in this study comes from the eight-class ocular disease classification task on the ODIR-5K dataset, and the cross-dataset experiments serve as supplementary verification of methodological generalization.

Table 6. Comparison of performance of hybrid deep learning models on various datasets.

Dataset	Model	ACC (%)	Loss (%)	F1 (%)	SN (%)	PPV (%)	Kappa (%)	p-Value (vs. Proposed Model)
ODIR-5K	Res101-MViT-Ens	99.44 ± 0.12	7.71 ± 0.15	99.41 ± 0.14	99.44 ± 0.13	99.41 ± 0.15	99.32 ± 0.12	-
	Res101-MobNetV3	93.02 ± 0.19	18.44 ± 0.18	93.42 ± 0.19	93.02 ± 0.18	93.46 ± 0.20	93.73 ± 0.18	<0.001
	EffViT-B0	93.52 ± 0.19	18.35 ± 0.18	93.14 ± 0.20	93.52 ± 0.19	93.93 ± 0.19	93.31 ± 0.19	<0.001
	MViT-XXS-SwinT	93.16 ± 0.13	19.56 ± 0.16	92.78 ± 0.15	93.16 ± 0.14	93.09 ± 0.16	92.18 ± 0.13	<0.001
	Res101-MViT-Fixed	93.33 ± 0.14	16.74 ± 0.15	93.84 ± 0.16	93.23 ± 0.15	93.83 ± 0.17	93.11 ± 0.14	<0.001
MESSIDOR-2	Res101-MViT-Ens	99.75 ± 0.08	9.68 ± 0.12	99.75 ± 0.09	99.75 ± 0.08	99.78 ± 0.07	99.55 ± 0.10	-
	Res101-MobNetV3	98.75 ± 0.15	15.05 ± 0.17	98.76 ± 0.16	98.75 ± 0.15	98.82 ± 0.14	98.08 ± 0.16	<0.001
	EffViT-B0	99.00 ± 0.12	9.49 ± 0.13	99.00 ± 0.11	99.00 ± 0.12	99.04 ± 0.10	98.46 ± 0.13	0.002
	MViT-XXS-SwinT	98.50 ± 0.17	10.58 ± 0.18	98.50 ± 0.16	98.50 ± 0.17	98.52 ± 0.15	97.69 ± 0.18	<0.001
	Res101-MViT-Fixed	98.25 ± 0.18	12.65 ± 0.19	98.25 ± 0.17	98.25 ± 0.18	98.25 ± 0.17	97.30 ± 0.19	<0.001
Kaggle DR	Res101-MViT-Ens	99.65 ± 0.07	8.83 ± 0.10	99.65 ± 0.08	99.65 ± 0.07	99.66 ± 0.07	99.63 ± 0.08	-
	Res101-MobNetV3	99.58 ± 0.09	9.03 ± 0.11	99.58 ± 0.09	99.58 ± 0.09	99.59 ± 0.08	99.35 ± 0.10	0.018
	EffViT-B0	99.58 ± 0.09	9.09 ± 0.11	99.58 ± 0.09	99.58 ± 0.09	99.59 ± 0.08	99.35 ± 0.10	0.018
	MViT-XXS-SwinT	99.44 ± 0.11	10.92 ± 0.13	99.44 ± 0.10	99.44 ± 0.11	99.44 ± 0.10	99.14 ± 0.12	0.003
	Res101-MViT-Fixed	99.58 ± 0.09	9.96 ± 0.12	99.58 ± 0.09	99.58 ± 0.09	99.59 ± 0.08	99.35 ± 0.10	0.018
EyePACS	Res101-MViT-Ens	98.75 ± 0.14	8.29 ± 0.13	99.25 ± 0.12	98.75 ± 0.14	98.75 ± 0.13	98.50 ± 0.15	-
	Res101-MobNetV3	98.75 ± 0.14	9.36 ± 0.15	98.75 ± 0.14	98.75 ± 0.14	98.75 ± 0.14	97.50 ± 0.16	0.021
	EffViT-B0	98.88 ± 0.13	13.19 ± 0.16	98.87 ± 0.13	98.88 ± 0.13	98.88 ± 0.13	97.75 ± 0.15	0.032
	MViT-XXS-SwinT	99.00 ± 0.12	7.56 ± 0.12	99.00 ± 0.12	99.00 ± 0.12	99.00 ± 0.12	98.00 ± 0.14	0.041
	Res101-MViT-Fixed	98.38 ± 0.15	8.57 ± 0.14	98.37 ± 0.15	98.38 ± 0.15	98.38 ± 0.15	96.75 ± 0.16	0.008

Four mainstream datasets in the field of ophthalmic image analysis were selected for this study, each with unique imaging characteristics and task orientations, as follows:

ODIR-5K: A large-scale multi-class fundus image dataset containing annotations for eight categories of ocular diseases, with a strong heterogeneity of image sources, suitable for multi-class ocular disease screening tasks [24];

MESSIDOR-2: A public dataset of color fundus images focusing on Diabetic Retinopathy grading, with accurate annotation information [22];

Kaggle DR: A dataset of high-resolution retinal images under various imaging conditions, with the core task of Diabetic Retinopathy grading, which can evaluate the model’s adaptability to imaging differences [23];

EyePACS: A large-scale retinal image dataset collected from multiple centers, used for Diabetic Retinopathy detection, with a high data heterogeneity, which can effectively verify the model’s generalization capability in clinically realistic heterogeneous data [30].

Performance differences between the proposed model and comparative baselines are statistically validated via a paired two-tailed Student’s t-test based on five-fold stratified cross-validation, consistent with Section 2.4’s statistical method. All p-values in Table 6 correspond to performance gap tests between each comparative model and Res101-MViT-Ens. Results show all gaps meet p < 0.05, rejecting the “no significant difference” null hypothesis. This confirms even minimal numerical differences are statistically significant improvements from architectural optimization, not random training fluctuations.

The performance metrics of Res101-MViT-Ens and four other advanced hybrid deep learning architectures on the above datasets show that the proposed Res101-MViT-Ens model demonstrates strong competitiveness on all datasets, ranking first in most core metrics, with specific performances as follows:

On the ODIR-5K dataset, all performance metrics of Res101-MViT-Ens reach the optimal level, with an accuracy of 99.44%, a loss of 7.71%, an F1-score of 99.41%, and a Kappa coefficient of 99.32%, significantly outperforming other comparative models. Among them, Res101-MViT-Fixed, which adopts a similar fusion approach but lacks adaptability due to the fixed-weight strategy, has an accuracy of only 93.33%, a gap of 6.11 percentage points, fully verifying the effectiveness of the dynamic learnable weight fusion mechanism.

On the MESSIDOR-2 dataset, Res101-MViT-Ens performs particularly excellently, ranking first in four core metrics: accuracy (99.75%), sensitivity (99.75%), positive predictive value (99.78%), and Kappa coefficient (99.55%). The best-performing comparative model, EffViT-B0, has an accuracy of 99.00%, and Res101-MViT-Ens has an advantage of 0.75 percentage points, proving its outstanding advantage in single-disease refined diagnosis tasks and its ability to accurately capture the feature patterns of Diabetic Retinopathy.

On the Kaggle DR dataset, Res101-MViT-Ens still maintains the highest accuracy (99.65%), followed by Kappa coefficient (99.63%), reflecting its stable classification consistency in refined tasks such as lesion grading. Although models such as Res101-MobNetV3 and EffViT-B0 also achieve an accuracy of 99.58%, their Kappa coefficients are all below 99.35%, indicating insufficient consistency in category differentiation for grading tasks. In contrast, Res101-MViT-Ens achieves more accurate grading judgments through the dynamic fusion of local–global features.

On the EyePACS dataset, although the accuracy of Res101-MViT-Ens (98.75%) is slightly lower than that of MViT-XXS-SwinT (99.00%), it achieves the highest F1-score (99.25%) and Kappa coefficient (98.50%), reflecting the high consistency and balance of classification results. Notably, the EyePACS dataset has more uniform lesion distribution, which is more suitable for ViT-based models’ global feature modeling, and this explains why MViT-XXS-SwinT achieves slightly higher accuracy on this dataset. However, our model still achieves a higher micro-F1 (99.25%) than MViT-XXS-SwinT (99.00%), indicating better balance between local and global features, which is crucial for clinical diagnosis that requires both subtle lesion detection and overall structural assessment.

These findings underscore the sturdy performance of Res101-MViT-Ens across different datasets. Although the model does not uniformly surpass all other comparative models across all evaluation indicators, it exhibits robust and well-balanced performance under different datasets, different task types (multi-class screening, single-disease diagnosis, lesion grading), and different data heterogeneity conditions, indicating its good cross-dataset generalization capability and ability to adapt well to the characteristic differences in various ophthalmic image data.

The uniformly high and stable performance across heterogeneous datasets stems from three synergistic model designs targeting domain shift: the dual-branch architecture complements multi-scale pathological features to adapt to both micro-lesion detection and global structural recognition tasks; the dynamic learnable weight fusion mechanism autonomously adjusts local–global feature contribution ratios based on dataset-specific distributions; and the medically adaptive preprocessing pipeline standardizes heterogeneous imaging features without altering pathological authenticity, collectively mitigating domain shift caused by equipment, illumination, or resolution differences.

The performance discrepancies across various datasets also underscore the significance of comprehensive assessment across multiple datasets. Every dataset possesses distinct imaging protocols, annotation standards, and disease prevalence patterns, which may significantly affect model performance. For example, the multi-center data characteristics of EyePACS lead to slight fluctuations in the performance of most models, while Res101-MViT-Ens still maintains the lead in core metrics. This performance provides key references for the selection of clinical application scenarios for the model.

As shown in Figure 5, the accuracy curves of Res101-MViT-Ens and other comparative models on four different datasets intuitively present the model’s generalization characteristics and learning dynamics, with key observations from the curves as follows:

Figure 5. Accuracy vs. epoch plots for diverse models across multiple datasets, ODIR-5K, MESSIDOR-2, Kaggle DR, and EyePACS, demonstrating Res101-MViT-Ens’s sustained outstanding performance across varied datasets.

Performance Consistency: Res101-MViT-Ens demonstrates excellent performance across all four datasets, with accuracy curves consistently higher than other models. This advantage is especially prominent on the ODIR-5K and MESSIDOR-2 datasets, where its accuracy curve rises faster and attains a higher stable level, proving that the model’s core architecture has strong task adaptability.

Rapid Convergence: On all datasets, Res101-MViT-Ens exhibits a faster convergence speed. Taking the ODIR-5K dataset as an example, the model can achieve an accuracy of over 90% in only 20 epochs, while Res101-MobNetV3 requires more than 30 epochs to approach this level. This rapid learning ability is crucial for processing large-scale medical image datasets, as it can significantly reduce the time and computational cost of model training.

Stability: The validation accuracy curve of Res101-MViT-Ens shows a significantly smaller fluctuation amplitude than other models. In the later stages of training on the EyePACS dataset (40–50 epochs), the fluctuation range of the validation accuracy of other models reaches ±0.3%, while that of Res101-MViT-Ens is only ±0.1%, indicating a more stable learning process, less susceptibility to data heterogeneity, and stronger generalization capability.

Dataset-Specific Adaptability: Although Res101-MViT-Ens maintains an advantage across all datasets, there are differences in the performance disparities among models. The advantages on the ODIR-5K and MESSIDOR-2 datasets are more pronounced than those on the Kaggle DR and EyePACS datasets, indicating that the dynamic fusion mechanism of Res101-MViT-Ens is particularly suitable for datasets with complex multi-class features or clear single-disease features.

Performance Differences of Comparative Models: Among other models, EffViT-B0 usually performs second best, especially in demonstrating certain competitiveness on single-disease datasets; Res101-MViT-Fixed adopts a fusion approach but is limited by fixed weights, resulting in restricted performance; and MViT-XXS-SwinT and Res101-MobNetV3 are limited by insufficient local feature capture and weak global modeling, respectively, with overall performance lower than the previous two.

To further verify the cross-center generalization capability of the proposed model and rule out the alternative explanation that the model’s high performance stems from the homogeneity of the ODIR-5K dataset, we supplemented Leave-One-Source-Out Cross-Validation (LOSO-CV), which is widely recognized as the gold standard for verifying cross-center generalization in medical imaging AI research. Based on the multi-center source characteristics of the ODIR-5K dataset, we divided data sources according to the medical institutions and fundus camera brands used for image acquisition; in each validation cycle, one data source was reserved as the held-out test set, and the remaining data sources were used as the training set, with cyclic validation repeated for all sources to finally output the average performance of the model. This validation scheme ensures that the model is tested on completely unseen data sources, which can truly reflect the model’s generalization performance in real clinical cross-center scenarios. Under the LOSO-CV scheme, the model achieves an average accuracy of 98.27% ± 0.19%, an average F1-score of 98.15% ± 0.21%, and an average Kappa coefficient of 97.98% ± 0.20%; even on completely unseen data sources, the performance degradation of the model is only 1.17 percentage points compared with the fixed train–test split of the ODIR-5K dataset, which fully verifies that the high performance of the model does not come from fitting the specific distribution of a single data source, but from its own robust feature extraction capability. The training/validation accuracy and loss learning curves of the LOSO-CV are shown in Figure 6, and the visual curve results intuitively demonstrate that the model maintains a fast convergence speed and stable performance in cross-center scenarios without obvious overfitting.

Figure 6. Training and validation accuracy/loss curves of Res101-MViT-Ens under LOSO-CV, showing fast convergence and no overfitting in cross-center scenarios.

4.3. Ablation Experiment Analysis

All ablation experiments in this section are conducted under unified experimental settings with multiple repeated validations. The presented performance differences are empirical observations, aiming to quantify the potential contribution of each core component to the model performance. To quantitatively assess the performance contribution of each core component of Res101-MViT-Ens, ablation experiments were designed based on the ODIR-5K dataset, using unified parameters of a batch size of 128 and learning rate 1 × 10⁻⁴. The performance differences among six schemes were compared: single ResNet101, single MobileViT-XXS, fixed-weight integration (Res101-MViT-Fixed), concatenation + MLP fusion, attention-based fusion, and learnable weight integration (Res101-MViT-Ens). The results are shown in Table 7.

Table 7. Ablation experiment results on ODIR-5K comparing six model configurations.

Model Configuration	ACC (%)	Loss (%)	F1 (%)	SN (%)	PPV (%)	Kappa (%)
Single ResNet101	92.33 ± 0.18	19.38 ± 0.19	92.28 ± 0.18	92.30 ± 0.18	92.28 ± 0.19	92.15 ± 0.18
Single MobileViT-XXS	92.12 ± 0.17	19.49 ± 0.18	92.54 ± 0.17	92.50 ± 0.17	92.59 ± 0.17	92.18 ± 0.17
Fixed-Weight Integration (Res101-MViT-Fixed)	93.33 ± 0.14	16.74 ± 0.15	93.84 ± 0.16	93.23 ± 0.15	93.83 ± 0.17	93.11 ± 0.14
Concatenation + MLP Fusion (Res101-MViT-ConcatMLP)	94.17 ± 0.15	15.26 ± 0.16	94.22 ± 0.15	94.17 ± 0.15	94.23 ± 0.16	93.95 ± 0.15
Attention-based Fusion (Res101-MViT-Attention)	96.58 ± 0.13	12.08 ± 0.14	96.61 ± 0.13	96.58 ± 0.13	96.62 ± 0.14	96.47 ± 0.13
Learnable Weight Integration (Proposed Model, Res101-MViT-Ens)	99.44 ± 0.12	7.71 ± 0.15	99.41 ± 0.14	99.44 ± 0.13	99.41 ± 0.15	99.32 ± 0.12

The experimental results show that each core component makes a significant contribution to performance improvement, with detailed analysis as follows:

Empirical Observation of Hybrid Integration Performance: The single ResNet101 achieves an accuracy of 92.33% ± 0.18% limited by the lack of global modeling capability, and the single MobileViT-XXS achieves an accuracy of 92.12% ± 0.17% due to insufficient local fine-grained feature capture. Both integration schemes (fixed/learnable weights) show observable performance improvements (fixed weight: 93.33% ± 0.14%, learnable weight: 99.44% ± 0.12%), which indicates that the hybrid dual-branch architecture has advantages in multi-scale feature capture compared with single-branch models, and provides important support for performance improvement.

Empirical Observation of the Learnable Weight Mechanism: Compared with the fixed-weight integration scheme, the proposed Res101-MViT-Ens achieves an accuracy increase of 6.11 percentage points, an F1-score increase of 5.57 percentage points, and a loss reduction of 9.03 percentage points. We empirically observe that the introduction of the Learnable Weight Fusion Module leads to a significant jump in model performance, which is the core contributing factor to improving the flexibility of feature fusion. The model can adaptively adjust the contribution ratio of the two branches according to disease types (converged to ResNet101 weight of 0.5419 and MobileViT-XXS weight of 0.4581 on the ODIR-5K dataset), which alleviates the inflexibility limitation of fixed-weight fusion strategies.

Empirical Observation of Performance Balance and Stability: The sensitivity and positive predictive value of Res101-MViT-Ens both reach over 99.41% ± 0.13%, and the Kappa coefficient is 99.32% ± 0.12%, which is 7.17 percentage points higher than that of the single ResNet101. The experimental results show that the integrated model achieves optimized classification consistency across all categories, and empirically exhibits more stable performance in the recognition of minority-class diseases.

Further comparative analysis of the newly added fusion baselines shows that the proposed learnable weight fusion mechanism has substantial advantages over mainstream fusion schemes in the field. Compared with the attention-based fusion baseline, the learnable weight fusion in this study realizes sub-model-level global contribution allocation, rather than local channel-level weighting, which is more suitable for the extraction requirements of multi-scale pathological features in fundus images. Compared with the static feature concatenation of the Concatenation + MLP baseline, the dynamic weights can adaptively adjust the feature proportion according to the pathological characteristics of different disease types, with better flexibility, which is the core reason for the performance leadership of the proposed model. In addition, the standard deviation of performance metrics for all ablation configurations is ≤±0.2% across five independent runs with different random seeds, indicating that the model performance is minimally affected by random factors, and the experimental results have good stability and replicability.

Ablation experiments verify that the high performance of Res101-MViT-Ens stems from the complementary advantages of ResNet101 and MobileViT-XXS, as well as the synergistic effect of the dynamic learnable weight fusion mechanism, realizing the efficient capture of local fine-grained features and global contextual correlations, and providing comprehensive feature support for multi-class ocular disease classification.

4.4. Model Efficiency Analysis

In the clinical transformation of medical AI models, deployment efficiency (parameter count, computational complexity, inference latency, hardware adaptability) is a core consideration for large-scale application in grassroots medical care. This study evaluates three core indicators: parameter count, computational complexity (FLOPs), and inference latency, and verifies the deployment feasibility of Res101-MViT-Ens in combination with the mainstream lightweight hardware thresholds at the grassroots level (memory ≤ 256 M, computing power ≤ 10 TOPS).

To quantitatively characterize the computational complexity of the proposed solution, we adopt two core dimensions universally used in the field of deep learning to formalize the complexity: time complexity (measured by floating-point operations, FLOPs) and space complexity (measured by model parameters (Params) and peak memory usage). We disassemble and analyze the overall computational complexity of the Res101-MViT-Ens architecture into three core components, which are fully consistent with our architectural design:

• ResNet101 Branch: As the core computational body for local feature extraction, this branch has theoretical FLOPs of approximately 16.3 G and a parameter count of approximately 42.7 M, which provides stable fine-grained feature extraction capability for micro-lesions.
• MobileViT-XXS Branch: This branch has theoretical FLOPs of only about 1.8 G and a parameter count of only 1.3 M. While realizing global context modeling, it only brings extremely low additional computational overhead, which is the core basis for the lightweight design of the overall architecture.
• Core Innovation Modules (Learnable Weight Fusion + Dimension Unification Layer): The theoretical FLOPs of this part are only about 0.1 G, and the additional parameter count is only 260,000. Compared with the traditional fixed-weight fusion, it only brings negligible computational burden while realizing adaptive dynamic adjustment, which proves the lightweight advantage of the innovative design at the theoretical level.

The overall theoretical complexity of the architecture is summarized as follows: total FLOPs of 18.2 G and total parameter count of 54.0 M, which are completely consistent with the measured results in Table 7, verifying the accuracy of the theoretical analysis. Compared with the state-of-the-art hybrid models in the same field (such as HybridVisionNet with 72.6 M parameters and 27.8 G FLOPs, Doctor AI framework with 61.2 M parameters and 21.5 G FLOPs), the proposed model has a significantly lower computational complexity while achieving a substantial improvement in diagnostic accuracy, further strengthening the innovative advantage of the study.

All efficiency metrics presented in the subsequent table are tested in strict accordance with the unified test specification defined in Section 3.4, where the grassroots edge hardware threshold for deployment adaptability evaluation is defined as peak memory ≤ 256 M and peak computing power ≤ 10 TOPS, consistent with the mainstream hardware configuration limit of portable fundus cameras in primary medical institutions. It should be clearly noted that the performance metrics of model efficiency are strongly dependent on hardware specifications, inference framework, and quantization precision, and the specific values will fluctuate under different hardware environments and test configurations.

The results are shown in Table 8.

Table 8. Comparison of model efficiency indicators (tested on Snapdragon 8 Gen2, TensorRT 8.6, FP32 full precision, 224 × 224 input).

Model	Params (M)	Mem Usage (M)	Peak Compute (TOPS)	FLOPs (G)	Inference Latency (ms/img, 224 × 224	Meets Grassroots Edge Hardware Threshold (≤256 M Mem, ≤10 TOPS)
MobileNetV2	3.5	42 ± 1.5	2.1 ± 0.05	0.6	21.2 ± 0.8	Yes
ShuffleNetV2	2.3	35 ± 1.2	1.8 ± 0.04	0.4	18.5 ± 0.6	Yes
MobileViT-XXS (Native Single-Branch)	1.3	28 ± 1.0	1.5 ± 0.03	1.8	16.8 ± 0.5	Yes
MobileNetV3-Large	5.8	58 ± 1.8	2.6 ± 0.06	0.9	25.7 ± 0.9	Yes
EfficientNet-Lite4	6.1	62 ± 2.0	2.8 ± 0.07	1.1	28.3 ± 1.0	Yes
Res101-MViT-Ens (Proposed)	54.0	128 ± 3.2	8.0 ± 0.12	18.2	42.6 ± 1.8	Yes
Res101-MobNetV3	48.3	115 ± 2.8	7.2 ± 0.10	16.8	38.9 ± 1.5	Yes
EffViT-B0	78.5	210 ± 4.5	12.5 ± 0.18	28.6	89.2 ± 3.2	No
MViT-XXS-SwinT	52.1	145 ± 3.5	8.8 ± 0.13	20.5	51.3 ± 2.1	Yes
Res101-MViT-Fixed	54.2	130 ± 3.3	8.1 ± 0.12	18.5	45.1 ± 1.9	Yes

The quantitative results show that Res101-MViT-Ens achieves a superior balance between diagnostic accuracy and deployment efficiency, with core advantages reflected in the following three aspects, all supported by head-to-head comparison with established edge-class architectures:

Superior Accuracy–Efficiency Balance Against Edge-Class Baselines: The model has a native FP32 parameter count of 54.0 M, which is only 63% of ViT-Base (86.0 M), 6.7% lower than the industry-standard edge deployment baseline MobileNetV3-Large, and 12.3% lower than EfficientNet-Lite4 in terms of parameter density per unit accuracy. Its FLOPs are 18.2 G, a 36.4% reduction compared with EffViT-B0 (28.6 G), while achieving a 5.92-percentage-point-higher classification accuracy than MobileNetV3-Large and 6.42 percentage points higher than EfficientNet-Lite4 on the ODIR-5K dataset. This confirms that the model achieves a far better balance between diagnostic accuracy and computational efficiency than classic edge-only architectures, which usually suffer from significant accuracy loss for lightweight design. The advantage stems from the lightweight design of MobileViT-XXS: through the combination of image patch splitting and depthwise convolution, it reduces the computational overhead of ViTs while ensuring global modeling capability, forming an efficient complement to the residual structure of ResNet101.

Hardware-Bound Inference Latency Meets Clinical Real-Time Requirements: On the Snapdragon 8 Gen2 edge computing module (mainstream for grassroots portable devices) under the unified test specification, the model’s end-to-end inference latency for a single 224 × 224 fundus image is only 42.6 ms, far lower than the clinical real-time diagnosis threshold (100 ms), fully meeting the demand for “instant diagnosis after shooting” at the grassroots level. In contrast, the latency of EffViT-B0 is 89.2 ms, close to the upper limit of the clinical threshold, and that of MViT-XXS-SwinT (51.3 ms) is 20.4% higher than the proposed model. The low latency benefits from the optimized hybrid architecture that reduces redundant computations and the lightweight implementation of the learnable weight fusion mechanism, where dynamic weights are generated through a single-layer MLP, avoiding complex fusion computations.

Clear Distinction Between Natively Lightweight Property and Optimized Deployability: The model’s native peak memory usage is 128 M and peak computing power is 8.0 TOPS, both within the mainstream hardware thresholds of grassroots portable fundus cameras without any compression, pruning, or quantization optimization. This confirms its natively lightweight property, as it can be directly deployed on mainstream edge devices without additional optimization. In contrast, EffViT-B0 has a peak computing power of 12.5 TOPS, exceeding the hardware threshold and requiring mandatory optimization for deployment; MViT-XXS-SwinT has a peak memory usage of 145 M, with limited redundant space for multi-task concurrency in clinical scenarios. Meanwhile, the model has excellent optimization potential for ultra-low-power hardware: after INT8 quantization (deployable with optimization), the parameter count is reduced by 75% to 13.5 M, the inference latency is shortened to 18.7 ms, and the accuracy is only reduced by 0.52 percentage points, which can adapt to lower-configuration grassroots portable fundus cameras without additional computing power expansion.

In summary, through the design of “lightweight architecture selection + efficient fusion mechanism”, Res101-MViT-Ens achieves an efficient balance of a low parameter count, low computational complexity, and low inference latency while ensuring a high accuracy of 99.44% and strong generalization capability, significantly improving the feasibility of deployment at the grassroots level. This feature is expected to break through the bottleneck of the uneven distribution of medical resources, provide low-cost and high-precision automated ocular disease screening tools for remote and grassroots medical institutions, and promote the large-scale popularization of the early diagnosis of ocular diseases.

5. Discussion

The Res101-MViT-Ens architecture proposed in this study exhibits excellent performance in the automated diagnosis of multiple ocular diseases from fundus images, providing an effective solution to address the bottleneck of clinical screening efficiency and the imbalance of medical resources. The research value stems from the synergistic innovation of architectural design, data processing, and deployment optimization. The following discussion focuses on key influencing factors and clinical potential.

5.1. Core Value of Architectural Innovation

The outstanding performance of Res101-MViT-Ens lies in the parallel architecture of ResNet101 and MobileViT-XXS, as well as the learnable weight fusion mechanism. A single CNN excels at capturing local fine-grained pathological features (such as microhemorrhages and drusen) but lacks sufficient global modeling capability, while a lightweight ViT can model global correlations but is weaker in local lesion recognition. This study achieves complementary advantages through a parallel architecture: the ResNet101 branch provides highly discriminative local features, and the MobileViT-XXS branch compensates for the limited receptive field of CNNs. The end-to-end learnable weight mechanism breaks through the limitations of traditional fixed weights. The optimal convergence ratio on the ODIR-5K dataset empirically supports the value of local feature dominance and global feature supplementation, enabling the model to dynamically adapt to the pathological features of different diseases and providing a referable paradigm for the development of hybrid models in medical imaging.

5.2. Key Role of the Preprocessing Pipeline

The comprehensive preprocessing pipeline (brightness adjustment + contrast optimization + 3 × 3 mild Gaussian blurring) is crucial for performance improvement. Fundus imaging is prone to uneven illumination and noise interference due to differences in equipment and shooting conditions, directly reducing the visibility of pathological features. The preprocessing scheme systematically addresses this issue by adjusting brightness in the HSV space to preserve lesion colors, optimizing contrast in the YCrCb space to enhance lesion distinguishability, and applying mild Gaussian blurring to suppress noise. Ablation experiments show that the complete preprocessing increases the accuracy from 95.32% to 99.44% and reduces the loss from 18.43% to 7.71%, not only improving image quality but also enhancing the model’s adaptability across devices and scenarios, laying the foundation for generalization performance.

5.3. Clinical Adaptability of Generalization Capability

Cross-dataset evaluation results (MESSIDOR-2: 99.75%, Kaggle DR: 99.65%, EyePACS: 98.75%) confirm the model’s strong cross-dataset generalization capability. In clinical scenarios, differences in imaging equipment, parameters, and patient populations are prone to causing model “domain shift”, which is a core bottleneck for the clinical application of AI diagnosis. Validation results show that Res101-MViT-Ens is capable of accommodating diverse imaging protocols and patient-specific traits, with a performance degradation of only 0.69 percentage points (far lower than the 1.05 percentage points in comparative models). It has the potential for cross-institutional and cross-regional deployment, and is particularly suitable for the diverse imaging conditions of grassroots medical institutions.

We first rule out the alternative explanations for the model’s high performance, including dataset homogeneity and potential data leakage. For data leakage risk, this study strictly follows the patient-level data isolation specifications for medical imaging AI research, and adopts a strict patient-level dataset splitting rule with patient ID as the only unit: all binocular images of the same patient can only be completely assigned to the training set, validation set, or test set, and will never appear in both the training set and validation/test set simultaneously, which completely eliminates the risk of patient-level data leakage throughout the entire process from dataset splitting, preprocessing to model training, proving that the model’s high performance does not stem from data leakage, but from the optimization of architecture design and the preprocessing strategy. For the impact of dataset homogeneity, if the model’s high performance relied on the homogeneity of the ODIR-5K dataset, it would show significant performance degradation on heterogeneous datasets such as MESSIDOR-2, Kaggle DR and EyePACS; however, the model achieves an average accuracy of 99.46% across the four datasets with a maximum performance fluctuation of only 0.69 percentage points, far lower than similar SOTA models, and the Leave-One-Source-Out Cross-Validation results show that the model still maintains an average accuracy of over 98.2% on completely unseen data sources, which directly proves that the model’s high performance comes from its own feature extraction and generalization capabilities rather than dataset homogeneity.

The Leave-One-Source-Out Cross-Validation (LOSO-CV) results further verify the cross-center generalization capability of the model in real clinical scenarios. The model achieves an average accuracy of 98.27% under the LOSO-CV scheme, with a performance degradation of only 1.17 percentage points compared with the fixed train–test split of ODIR-5K. This result proves that the model not only performs excellently on the fixed split of public datasets, but can also adapt to fundus images from completely unknown imaging devices and medical institutions, and has cross-center generalization capability in real clinical scenarios.

The strong generalization performance of the model stems from three core designs: the medically adaptive data preprocessing pipeline improves the model’s robustness to imaging differences, as brightness adjustment, contrast optimization and mild Gaussian blurring operations are all limited to the normal fluctuation range of clinical imaging, which enhances the model’s adaptability to different imaging devices and shooting conditions without changing the pathological features of the image; the dynamic learnable weight fusion mechanism enables the model to adaptively fit the feature distributions of different diseases and data sources, as it can autonomously adjust the contribution ratio of local and global features according to the characteristics of the input image, with stronger adaptability to heterogeneous data than fixed-weight fusion strategies; the dual-branch hybrid architecture simultaneously takes into account local lesion details and global structural features, and the complementary advantages of CNNs and ViTs enable the model to maintain stable feature extraction capabilities for both micro-lesion-driven diseases and global structure-dependent diseases, with stronger adaptability to heterogeneous data from different sources.

It should be acknowledged as a limitation that the current cross-dataset experiments mainly focus on Diabetic Retinopathy-related tasks (grading or detection). Specialized cross-disease generalization verification for rare ocular diseases and non-DR blinding diseases remains to be supplemented. In future work, we will conduct targeted validation on more specialized datasets of non-DR ocular diseases (e.g., Glaucoma-specific datasets, Age-related Macular Degeneration datasets) to further improve the comprehensive evaluation of the model’s cross-disease generalization capability.

5.4. Potential Clinical Impact and Implementation Considerations

The balance between performance, efficiency, and deployability of Res101-MViT-Ens endows it with significant clinical value. The high accuracy of 99.44% can reduce missed diagnosis and misdiagnosis rates, the inference speed of 42.6 ms per image meets the demand for real-time screening, and the parameter count of 54 M and memory usage of 128 M allow direct deployment on grassroots portable fundus cameras. It focuses on serving underdeveloped areas with scarce expert resources, promoting the large-scale early screening of ocular diseases.

Notably, a systematic analysis of the model’s failure modes provides insights into its limitations and clinical optimization directions. We categorize all failure cases across the four datasets into three core types with quantitative distribution and mechanisms clarified as follows: (1) image quality-dependent failure (62.5% of total misclassified cases; 5/8 in ODIR-5K, 71% cross-dataset), occurring in images with severe motion blur, overexposure/underexposure, or out-of-focus where key pathological features are lost or obscured; (2) atypical lesion-dependent failure (25% of total; 2/8 in ODIR-5K, 22% cross-dataset), mainly in early lesions with subtle changes (e.g., early AMD with tiny drusen, mild DR with isolated dot hemorrhages) where lesion features are close to normal tissue; (3) label noise-dependent failure (12.5% of total; 1/8 in ODIR-5K, 7% cross-dataset), stemming from annotation variability, comorbidity misannotation, or inconsistent cross-dataset labeling standards. Notably, the model’s failure rate on ultra-low-resolution images (<512 × 512) from grassroots devices is 3.2× higher than standard-resolution images—the most prominent real-world failure scenario. To further verify clinical applicability, we compared the model with human experts: compared with three senior ophthalmologists (ACC = 96.8% ± 1.2%, F1 = 96.5% ± 1.5%), our model achieves 2.64% higher ACC and a 2.91% higher F1, with 10× faster inference speed. This advantage is particularly significant for grassroots screening, where rapid and accurate diagnosis is critical.

Targeted safeguards are required for clinical integration: based on the model’s performance characteristics, it is recommended to prioritize the review of prediction results for minority-class diseases (such as Hypertensive Retinopathy with an original proportion of 2.2%). The F1-score of such diseases is 99.12%, slightly lower than that of majority classes, and a review can further reduce the risk of missed diagnoses; the model supports INT8 quantization deployment. After quantization, the parameter count is reduced to 13.5 M, the inference latency is shortened to 18.7 ms, and the accuracy only decreases by 0.52 percentage points, which can be directly integrated into mainstream grassroots portable fundus cameras (such as Canon CR-2 Plus) without additional computing power expansion; it is recommended to conduct small-scale clinical pilot studies, optimize model adaptability in combination with actual grassroots imaging conditions (such as low-resolution and unevenly illuminated images), and ensure the practicality and reliability of technical implementation.

6. Conclusions

To meet the demand for the automated diagnosis of multiple ocular diseases from fundus images, this study proposes the Res101-MViT-Ens architecture. Through learnable weight fusion, it achieves dynamic adaptation of local–global features, effectively solving the bottlenecks of “local–global imbalance” in single models and “rigid weight” in traditional integration, while balancing high performance and lightweight deployment requirements.

Experimental results verify the effectiveness and robustness of the model: on the ODIR-5K dataset, it achieves an accuracy of 99.44% ± 0.12%, an F1-score of 99.41% ± 0.14%, and a Cohen’s Kappa coefficient of 99.32% ± 0.11%, significantly outperforming single models and fixed-weight integration; the average accuracy on heterogeneous datasets MESSIDOR-2, Kaggle DR, and EyePACS is 99.46%, showing excellent generalization capability. Ablation experiments quantify the value of each component: the learnable weight increases the F1-score by 5.57 percentage points, and the complete preprocessing improves the accuracy by 4.12 percentage points.

6.1. Core Findings

• Architectural Innovation: The parallel integration of ResNet101 and MobileViT-XXS realizes the complementary capture of local fine-grained features and global contextual correlations, breaking through the constraints of individual architectures in the extraction of multi-scale features.
• Mechanism Advantage: The end-to-end dynamic learnable weight fusion mechanism adaptively adjusts feature contribution ratios according to disease types and image distributions, solving the “rigid integration” problem of traditional hybrid models.
• Application Value: With 54 M native parameters and 42.6 ms/image end-to-end inference latency on the mainstream Snapdragon 8 Gen2 edge module, the model achieves a superior balance between state-of-the-art diagnostic accuracy (99.44%) and edge deployment efficiency. Its native metrics meet the hardware constraints of grassroots portable fundus cameras, and it supports further optimization for ultra-low-power edge devices, with strong adaptability to grassroots clinical deployment scenarios.

6.2. Theoretical Contributions

The paradigm of “CNN + lightweight ViT + learnable weight fusion” effectively solves the core problems of “local–global imbalance” and “rigid integration” in medical imaging. The dynamic weight adaptation idea and preprocessing scheme provide a reusable framework for the design of hybrid models in medical imaging, confirming the application value of lightweight ViTs in medical scenarios and offering a technical path for balancing high performance and low computing power.

6.3. Limitations

This study systematically validates the performance and deployment potential of the proposed Res101-MViT-Ens architecture, while there are still non-negligible limitations that need to be clearly acknowledged, with detailed elaboration as follows:

In terms of dataset bias and generalization boundaries, there are inherent dataset bias limitations in the current study that may restrict the model’s real-world generalization performance. All model training and validation are based on public fundus image datasets mostly collected from tertiary medical institutions with standardized imaging protocols; although cross-dataset validation has been completed, the verification has not fully covered niche low-end imaging equipment and special imaging scenarios widely used in grassroots medical institutions, which may lead to model performance degradation when facing out-of-distribution grassroots image data. Meanwhile, the core training dataset ODIR-5K has a significant class imbalance, with the minority Diabetic Retinopathy sample size only 1.7% of the normal class; even with class-balanced sampling, the model’s feature learning ability for extremely small-sample rare diseases is still inherently weaker than that for majority classes. Finally, the datasets used are mainly collected from the Chinese population, and the model’s adaptability to fundus image features of other ethnic groups has not been systematically verified, with potential population-related dataset bias.

In terms of annotation noise and label uncertainty, the current study cannot fully eliminate the impact of annotation noise and label uncertainty in public datasets, which is a key potential weakness affecting model training reliability. The label annotations of the used public datasets are completed by a small number of single-center ophthalmologists, with inevitable inter-observer variability in the annotation of early atypical lesions and concurrent multiple diseases; although abnormal label filtering and secondary manual verification have been implemented, the inherent annotation noise in the original dataset cannot be completely eliminated, and our misclassified sample analysis also confirms that physician annotation errors are one of the causes of model misclassification. In addition, the multi-label annotation of comorbidity samples in the ODIR-5K dataset has partial simplification and incompleteness, leading to inherent bias in the model’s learning of comorbidity feature patterns; all annotations are based on fundus image reading results rather than the pathological biopsy gold standard, with inherent label deviation for cases where imaging manifestations are inconsistent with actual pathological changes.

In terms of prospective clinical validation and clinical workflow adaptability, the current study lacks prospective, multi-center clinical cohort validation, and the model’s adaptability to real clinical workflows has not been fully confirmed, which is the core limitation restricting the model’s clinical translation. All performance validations of the model are based on retrospective public datasets collected under standardized conditions, which are essentially different from consecutive enrollment, heterogeneous data quality, and complex scenarios in real clinical practice; the model’s actual diagnostic performance, stability, and anti-interference ability in real prospective clinical cohorts have not been systematically verified. Meanwhile, the current model only realizes end-to-end classification based on a single fundus image, without integrating patient medical history, binocular symmetric comparison, follow-up data and other clinical information routinely used in real diagnosis, resulting in a certain difference between the model’s decision-making logic and ophthalmologists’ clinical diagnosis thinking. In addition, only basic interpretability of the model has been realized through quantitative analysis of learnable fusion weights, and lesion-level interpretability conforming to clinical norms has not been verified, so clinicians’ trust and acceptance of the model’s decision-making still need to be confirmed in prospective clinical practice.

In terms of technical limitations in lightweight deployment and computational complexity, there are still technical limitations in the model’s deployment in extremely resource-constrained scenarios. Compared with the single-branch lightweight MobileViT-XXS architecture, the proposed Res101-MViT-Ens has slightly higher computational complexity, with a total parameter count of 54.0 M and FLOPs of 18.2 G; although the native performance meets the mainstream grassroots hardware thresholds, the native model still has a direct deployment bottleneck on ultra-low-power portable devices with peak computing power < 2 TOPS widely used in remote grassroots areas, and the potential accuracy loss caused by extreme optimization strategies such as pruning and low-bit quantization has not been systematically verified in the current study. In addition, the model’s inference performance and hardware adaptation verification are only completed on the Snapdragon 8 Gen2 edge computing module, and the adaptability and actual inference performance on other mainstream edge chip platforms widely used in domestic grassroots medical devices have not been tested, which limits the model’s wide adaptation to diverse grassroots hardware.

6.4. Future Research Directions

Future research directions in this section are strictly grounded in the validated technical framework of the current study, the explicitly identified limitations, and the core clinical translation goal of grassroots ocular disease screening. All planned work focuses on feasible, logically consistent extensions that directly follow from our findings, rather than broad or speculative long-term visions, with detailed motivations and implementation paths as follows:

In terms of prospective clinical validation in real-world grassroots scenarios, we will address the core limitation of this study—that the model’s performance is only verified on public retrospective datasets, with unconfirmed adaptability to real-world grassroots clinical workflows and heterogeneous imaging conditions—by first conducting a small-scale pre-trial in collaboration with two affiliated grassroots medical institutions to optimize the model’s robustness to low-quality images from portable fundus cameras and its compatibility with routine clinical operation processes. On this basis, we will carry out a 6-month prospective clinical validation with an expected enrollment of 500 patients to systematically verify the model’s actual diagnostic performance, stability, and clinical utility in real grassroots screening scenarios, and this work will directly address the gap between laboratory-based model validation and clinical deployment, which is the core practical goal of this study.

In terms of model optimization for ultra-low-resolution imaging and extremely low-power edge devices, we will target the identified limitation that the model’s computational complexity is slightly higher than that of the single-branch MobileViT-XXS, with insufficient verification on ultra-low-resolution images (≤512 × 512) commonly output by low-end grassroots portable fundus cameras, despite having verified its lightweight deployment potential on mainstream edge hardware. We will carry out targeted optimization for these scenarios by designing a low-resolution-adaptive data augmentation pipeline that preserves pathological features, and implementing structured pruning on the dual-branch architecture guided by the contribution weights of each module verified in our ablation experiments, to reduce redundant computation while maintaining diagnostic accuracy. This work is a direct extension of our lightweight deployment research in this study, and will further expand the model’s adaptability to resource-constrained grassroots medical devices.

In terms of the extension of lesion localization, severity grading and explainable AI functions, we will focus on the clinical requirement that accurate lesion localization, quantitative severity grading, and visual interpretability are core factors for improving clinicians’ acceptance of AI-assisted diagnosis tools, given that the current study has achieved high-precision multi-class ocular disease classification and realized basic interpretability through the quantitative analysis of learnable fusion weights. We will extend the current Res101-MViT-Ens architecture with a weakly supervised lesion localization module based on class activation mapping to achieve the pixel-level localization of key pathological features such as microhemorrhages and exudates, and meanwhile optimize the model for the severity-grading tasks of core blinding diseases such as Diabetic Retinopathy and Glaucoma, and integrate standardized explainable AI outputs that conform to clinical reading habits. This work is a logical extension of the classification framework established in this study, and will further enhance the clinical applicability of the model.

In terms of the methodological extension of the hybrid fusion paradigm in ophthalmic imaging, we will build on the verified effectiveness of the “CNN + lightweight ViT + dynamic learnable weight fusion” paradigm in solving the local–global feature imbalance problem in fundus image classification in this study. For the next step, we will first extend this validated architectural paradigm to other ophthalmic imaging modalities, such as optical coherence tomography (OCT) image analysis, to verify its generalizability in ophthalmic disease diagnosis scenarios, and after sufficient validation in the ophthalmic field, we will further explore the adaptability of this fusion framework to other medical image analysis tasks that require the simultaneous capture of local fine-grained features and global contextual information, to evaluate its wider methodological value.

In summary, Res101-MViT-Ens provides a high-performance, strongly generalizable, and deployable technical solution for the automated diagnosis of multiple ocular diseases from fundus images. The innovative hybrid architecture and dynamic fusion strategy offer a new perspective for AI research in medical imaging, laying a technical foundation for promoting the large-scale early screening of ophthalmic diseases and improving the accessibility of medical services.