RAGL-MR: Reliability-Aware Global-Local Metric Retrieval for Optical Image-Based Texture and Material Surface Recognition

Abstract

Optical image-based texture and material surface recognition is important for computational imaging, intelligent inspection, and material-reference retrieval, where a system often needs not only a closed-set label but also inspectable evidence and an expandable gallery. Scale-varying micro-textures, repeated surface structures, global layout cues, and visually ambiguous categories make this task difficult under non-controlled imaging conditions. This paper presents Reliability-Aware Global-Local Metric Retrieval (RAGL-MR), which maps images and patches into a shared ArcFace metric space, retrieves multi-scale patch neighbors as local surface evidence, compresses whole-image reference galleries using Multi-Prototype Global Distribution Modeling (MP-GDM), and combines branch scores through validation-selected reliability-aware fusion. The framework is evaluated on public texture/material surface image benchmarks used as proxies for surface image analysis rather than as controlled photonic experiments. On the first five official DTD splits, the final fusion reaches 74.06 ± 0.11 Top-1, 92.61 ± 0.61 Top-5, and 96.11 ± 0.23 Top-10; on MINC-2500, it reaches 85.84 ± 0.25 Top-1, 97.63 ± 0.28 Top-5, and 99.15 ± 0.10 Top-10, with additional checks on FMD and KTH-TIPS2-b. Additional perturbation, calibration, large-gallery, and industrial surface-defect checks further examine robustness, reliability, and practical retrieval behavior. These results indicate that RAGL-MR provides an evidence-aware and gallery-extensible retrieval complement for optical material surface image analysis.

1. Introduction

Optical imaging, photonic visual sensing, and computational imaging increasingly support surface inspection, material reference search, and intelligent material analysis. In these applications, surface texture, reflectance, microstructure, and global pattern organization are often as important as the object category itself, because they provide evidence about the observed material and the reliability of the visual decision.

Surface appearance in optical images is affected by illumination, viewpoint, scale, roughness, specular reflection, local micro-patterns, and larger layout structures. Related optical imaging work has long shown that changing illumination and view can reveal surface shape, reflectance, and material cues that are not captured by one image-level descriptor alone [1,2,3]. A single closed-set label or a single image-level feature can, therefore, hide the evidence needed for human verification, while one local scale can miss structures that appear at different receptive fields. Public texture/material image benchmarks help study this problem in a reproducible way: DTD emphasizes describable texture attributes collected in the wild [4], MINC-2500 provides a balanced subset of material images derived from Materials in Context [5], and FMD and KTH-TIPS2-b provide complementary material and controlled-texture settings [6,7]. These datasets are not controlled photonic-acquisition datasets, but they are useful proxies for evaluating scale variation, appearance diversity, and ambiguous local evidence in surface image analysis.

Earlier texture-recognition methods relied on hand-crafted local descriptors and statistical texture representations. Local binary patterns captured repeated local contrast patterns [8], filter-bank representations described response distributions [9], and bilinear pooling showed that feature interactions are effective for fine-grained visual discrimination [10]. Deep texture encoders such as Deep-TEN and DEP then learned task-specific texture manifolds [11,12]. Modern backbones and foundation features, including ConvNeXt, Vision Transformers, CLIP, DINO, and DINOv2, further improve image-level representation quality [13,14,15,16,17]. However, stronger features alone do not fully address the operational needs of optical inspection and material-image retrieval, where users may need a Top-5 or Top-10 candidate shortlist, visible neighbor evidence, gallery updates after training, and compact reference storage.

Existing solutions leave three practical issues insufficiently addressed: (1) closed-set classifiers and foundation backbones can be strong in Top-1 recognition, but their classifier heads or image-level neighborhoods often hide neighbor-level evidence and make post-training gallery updates inconvenient; (2) single-scale local analysis is insufficient for optical material surface images because fine micro-textures, medium repeated structures, and larger layout patterns may all be discriminative under illumination and scale variation and (3) full-gallery retrieval preserves evidence but can be redundant, whereas over-compressed prototypes may lose multi-modal material appearances, and global-local fusion needs validation-selected reliability diagnostics when optical surface patterns are ambiguous.

Motivated by these issues, this paper proposes Reliability-Aware Global-Local Metric Retrieval (RAGL-MR) for optical image-based texture and material surface recognition. The framework trains an ArcFace metric backbone and represents a query through three coordinated evidence sources: multi-scale local surface neighbors, compact global prototypes, and reliability-aware fusion cues. The goal is not to claim a universal Top-1 SOTA classifier, but to provide an evidence-aware, gallery-extensible, and reliability-diagnostic retrieval framework for optical material surface image analysis.

The main contributions are summarized as follows:

• Multi-scale local metric retrieval for optical surface evidence. We design a patch-level retrieval branch that extracts overlapping local surface evidence at multiple receptive fields and aggregates nearest-neighbor class votes, improving shortlist reliability for scale-varying texture and material surface patterns.
• Multi-Prototype Global Distribution Modeling (MP-GDM) for compact material surface galleries. We introduce MP-GDM to compress whole-image reference-gallery embeddings into multiple class prototypes, preserving intra-class appearance modes while reducing the memory and latency of global retrieval.
• Validation-selected reliability-aware fusion and systematic diagnostics. We formulate static, confidence-only, and disagreement-aware fusion candidates under a validation-controlled protocol, and evaluate them through overall comparison, ablation, confidence-bin diagnostics, calibration curves, branch-disagreement recovery, paired fix/break analysis, large-gallery scaling, perturbation robustness, runtime measurement, cross-dataset checks on DTD, MINC-2500, FMD, and KTH-TIPS2-b, and an industrial surface-defect sanity check.

2. Related Work

2.1. Optical Texture and Material Surface Image Recognition

Optical surface inspection and reflectance-oriented imaging provide a complementary motivation. Photometric stereo estimates surface orientation from multiple illuminations [1], polynomial texture mapping captures surface appearance under changing light directions [2], and differential angular imaging has been used to improve material recognition from angularly varying image evidence [3]. These works illustrate why surface texture, roughness, reflectance, and micro-patterns are important cues in image-based sensing. RAGL-MR does not introduce new optical hardware; instead, it organizes image evidence from public benchmarks into an inspectable retrieval framework suitable for later controlled optical validation.

Classical texture recognition used local binary patterns to encode repeated local contrast patterns [8]. Filter-bank and deep-filter-bank approaches then represented textures through response statistics over many local filters [9]. Bilinear pooling showed that second-order feature interactions are valuable for fine-grained visual recognition [10], while Deep-TEN and DEP learned task-specific texture encodings and manifolds [11,12]. These methods remain relevant to optical surface analysis because they treat local surface patterns as visual evidence, but most are optimized as closed-set recognizers rather than as expandable retrieval systems with inspectable patch-level support.

Modern image encoders provide stronger starting representations for image-based surface recognition but do not by themselves define an evidence-aware retrieval system. ConvNeXt and Vision Transformer represent the convolutional and patch-token families used in contemporary recognition [13,14], while Swin, BEiT, and MAE improve hierarchical modeling or masked pretraining [18,19,20]. Hybrid and efficient designs, including UniFormer, MetaFormer, InternImage, and EfficientViT, further explore the accuracy–efficiency trade-off [21,22,23,24]. Gated convolutional networks with hybrid connectivity also combine local residual reuse, global dense connectivity, and multi-kernel gated feature extraction for image classification [25]. These architectures improve feature representation and global-local feature extraction inside the backbone. In contrast, RAGL-MR does not propose a new backbone; it organizes an existing metric embedding into patch-neighbor evidence, prototype-compressed global retrieval, and reliability-aware Top-k output. Their main contribution is image-level representation quality; they do not directly specify how to expose local optical surface evidence, update a labeled material gallery after training, or maintain a reliable Top-k shortlist.

Foundation and self-supervised features offer strong transfer and retrieval potential for optical image understanding. CLIP learns transferable visual-language representations [15], whereas DINO and DINOv2 learn discriminative nearest-neighbor structure through self-supervision [16,17]. Retrieval analysis of these features confirms the value of their image-level neighborhoods [26]. RAGL-MR, therefore, includes foundation-feature controls, but addresses a different systems question: how local surface neighbors, compressed global evidence, and reliability cues can be organized around an expandable material surface reference gallery.

2.2. Multi-Scale Local Evidence in Computational Imaging

Optical material surface recognition requires robustness to object context, illumination, scale, surface composition, and reflectance-related appearance changes. DTD and MINC offer complementary views of in-the-wild texture and material recognition [4,5], while FMD and KTH-TIPS2-b provide settings with different acquisition assumptions [6,7]. Recent methods also revisit how local evidence should be aggregated: MatSynth expands modern physically based material resources [27], RADAM uses randomized aggregation of deep activation maps [28], and GraphTEN models local/global texture-primitive relations and cross-scale dependencies [29]. These developments motivate representations that remain robust across optical appearance and scale, but they do not directly provide inspectable metric-retrieval evidence.

Classic multi-scale and structural transforms provide another important reference point. Ridgelets and curvelets were designed to capture line- and curve-like image singularities across scales [30,31,32], while phase congruency detects perceptually meaningful image features by locating phase agreement across frequency components [33,34]. These methods are interpretable and useful for denoising, edge/structure extraction, and handcrafted texture analysis. They are not adopted as the main representation here because RAGL-MR targets a different operating point: metric-learning-based reference-gallery retrieval with patch-neighbor evidence, prototype-compressed global search, and reliability diagnostics. Nevertheless, they clarify why multi-scale structure remains valuable for optical surface analysis.

Multi-scale evidence is also common in optical inspection, where fine surface roughness and larger repeated structures can appear simultaneously in the same image. Rather than modeling a particular sensor or illumination rig, the present work evaluates this idea as image-analysis evidence: multiple patch scales provide complementary local views before the results are fused into a ranked shortlist.

Metric learning optimizes an embedding space where nearest neighbors carry semantic meaning. Margin-based objectives such as ArcFace and CosFace strengthen class separation and are widely used for retrieval-oriented recognition [35,36]. In an optical material surface recognition setting, this formulation is attractive because a prediction can be supported by retrieved surface-image neighbors rather than only by a classifier logit. It also allows the reference gallery to be updated when new labeled material surface samples become available.

For texture/material surface images, local patches are especially important because a category can be defined by small reflectance or micro-texture fragments rather than the entire object. However, single-scale retrieval can be brittle when the discriminative pattern is finer or coarser than the selected patch size. Multi-scale analysis has, therefore, become a natural strategy in computational imaging, fine-grained recognition, and visual matching, but it also introduces noisy local evidence. This work treats multi-scale retrieval as a ranked-evidence module and evaluates whether it improves the candidate shortlist under a controlled validation protocol. The emphasis is different from ordinary multi-scale classification: the goal is not only to increase Top-1, but also to preserve the correct class among the top candidates for optical inspection or human verification.

2.3. Metric Retrieval and Prototype Compression for Expandable Surface Galleries

Prototype-based methods summarize classes with representative embeddings and have been used in few-shot recognition [37]. Efficient clustering and similarity search make prototype or compressed-gallery retrieval practical at larger scale [38,39]. A single prototype can be too coarse for material surfaces with multiple appearances, while a full reference gallery can be expensive and redundant. MP-GDM follows the prototype idea but keeps multiple class prototypes to preserve intra-class modes. This is particularly useful for material surface datasets, where the same class can contain different objects, illumination conditions, reflectance responses, and surface layouts.

The retrieval setting also changes how prototypes should be judged. In closed-set classification, a prototype is often treated as a compact decision representative [37]. In reference-gallery retrieval, however, prototypes must preserve enough visual diversity to keep the correct class in the ranked shortlist while still reducing search cost [39]. This tension is visible in material surface categories whose samples form several appearance clusters. A single centroid can lie between modes and match no real surface well, whereas too many retained gallery vectors reduce the benefit of compression. MP-GDM, therefore, treats prototype count and aggregation as validation-selected design choices rather than fixed constants.

Compact retrieval methods span a continuum from full memory banks to heavily compressed codebooks. A full bank preserves sample-level diversity but scales linearly in storage and similarity computation, whereas clustering and approximate search reduce redundancy at the cost of representation detail [38,39]. Multi-prototype banks occupy the useful middle ground: clustering removes redundant vectors while retaining several representative modes per class. Their effectiveness depends not only on prototype count, but also on how query-to-prototype similarities are aggregated; maximum aggregation favors the closest mode, whereas mean or smooth log-mean-exp aggregation incorporates broader class support. These design choices motivate the validation-selected prototype count and aggregation rule used by MP-GDM.

Prototype compression is also connected to reference-gallery maintenance. New labeled optical surface samples can be embedded and assigned to an existing mode, or can trigger prototype refresh without retraining the feature extractor. This property differs from replacing the final layer of a closed-set classifier and is useful when a material reference collection evolves over time. The present study evaluates this benefit through vector count, memory, latency, and ranked retrieval accuracy rather than treating compression as an implementation detail.

2.4. Reliability-Aware Fusion for Optical Image Understanding

Reliability-aware prediction is related to calibration and uncertainty estimation. Temperature-style calibration highlights that neural confidence can be misaligned with correctness [40], while Bayesian and ensemble-based approaches estimate predictive uncertainty from different modeling assumptions [41,42]. Test-time adaptation also uses entropy-related signals to adjust predictions under distribution shift [43]. In this paper, however, the goal is operational rather than purely probabilistic: confidence and branch agreement are used to decide whether local, multi-scale, or global optical surface evidence should dominate the ranked list. Because branch scores are not inherently calibrated across evidence sources, all fusion choices are selected on validation data. This design avoids test-set tuning and makes it possible to report not only the selected final model but also the reliability diagnostics that explain when fusion helps.

Global-local fusion is particularly delicate for optical texture/material images. A local branch may be confident because repeated surface fragments dominate the image, yet those fragments can also appear in a different class. A global branch may recover layout-level material evidence, but it may also be biased by object context, illumination, or background. Branch disagreement is, therefore, not a direct error signal; it is an ambiguity signal that should be interpreted together with confidence margins and validation behavior [40,43]. This is why RAGL-MR reports both final selected fusion and diagnostic fixed/broken cases rather than claiming that adaptive fusion always wins.

This ambiguity is more pronounced in material surface retrieval than in many object-recognition settings. Texture fragments such as stripes, fibers, pores, and folds can be locally convincing across several categories, while the whole image can contain context that is informative but not intrinsic to the material surface. Consequently, confidence values from local voting and global prototype similarity have different statistical meanings and cannot be combined safely without validation. Margin and entropy summarize within-branch certainty [40,43], while agreement reveals whether independent evidence sources support the same candidate. Their role is diagnostic: they identify cases in which fusion has an opportunity to recover an error, but they do not guarantee improvement for every disagreement case.

Reliability-aware fusion must, therefore, satisfy two constraints. It should preserve confident correct predictions instead of perturbing them unnecessarily, and it should improve ambiguous ranked lists without selecting weights on the test set. RAGL-MR addresses these constraints with validation-selected static priors and reliability-aware candidate rules, followed by confidence-bin, disagreement, and paired fixed/broken analyses. This operational view links calibration concepts to an inspectable retrieval outcome rather than reducing reliability to a single expected-calibration statistic.

Overall, the related literature provides strong ingredients—modern backbones, local optical evidence, prototype compression, and uncertainty cues—but these ingredients are usually studied separately. RAGL-MR connects them in a metric retrieval protocol where patch-level surface evidence, global prototype evidence, gallery extensibility, and reliability diagnostics are evaluated together. This positioning is deliberately narrower than generic material-classification SOTA, but it better matches optical image analysis scenarios where users need a ranked, inspectable, and updateable material surface shortlist.

3. Materials and Methods

3.1. Overview of the RAGL-MR Framework

The proposed framework receives a query optical material surface image and a labeled reference gallery, then produces a ranked class list. The design follows the three problems identified in the Introduction. To address the lack of inspectable evidence, the model is formulated as metric retrieval instead of a pure classifier head. To address scale mismatch, the local branch extracts surface patches at several granularities and aggregates nearest-neighbor class votes. To address unstable branch fusion, the final decision is selected from a validation-controlled family of static and reliability-aware fusion rules.

Figure 1 provides a data-driven overview of the full pipeline, intermediate evidence, and final ranking behavior. During training, images or patches are mapped into a normalized ArcFace embedding space. During evaluation, the query is processed by three evidence branches. The single-scale local branch provides a controlled patch-retrieval baseline. The multi-scale local branch expands the local receptive field set and improves candidate coverage through real patch-level crops and reference-gallery evidence. The MP-GDM branch uses whole-image embeddings but stores them as compact class prototypes rather than as a full reference gallery, which is visualized through an embedding map and retained prototype centers. The final module combines branch score distributions according to confidence, reliability, and disagreement cues to produce a Top-k material surface shortlist and reliability-oriented diagnostic outcome.

Figure 1. Overall RAGL-MR framework for optical material surface image retrieval. A query image and an expandable labeled reference gallery are mapped into a shared normalized metric space. Multi-scale local retrieval exposes patch-level surface evidence, MP-GDM compresses gallery embeddings into class prototypes, and reliability-aware fusion combines branch scores using validation-selected priors and confidence/agreement cues to produce an evidence-aware Top-k shortlist. Dashed module colors distinguish the input, embedding, local-retrieval, prototype, fusion, and output stages.

The six visual stages in Figure 1 correspond to the complete inference path: a query and expandable surface reference gallery enter the shared normalized metric space; multi-scale local retrieval exposes patch neighbors; MP-GDM replaces redundant whole-image vectors with retained prototypes and validation-selected fusion converts local, prototype, confidence, and agreement evidence into a ranked system output. The displayed output card summarizes the Top-1 prediction, Top-5 shortlist, ground-truth rank, and the evidence sources supporting the result.

The embedding backbone is shared by the global and local branches. For an input crop or image u, the network produces a feature $h_{\theta }\left(u\right)$, which is projected and normalized as

$$z\left(u\right)={\displaystyle \frac{h_{\theta }\left(u\right)}{{\parallel h_{\theta }\left(u\right)\parallel }_2}}.$$

(1)

Let $W_c$ be the normalized classifier weight of class c, and let ${\theta }_c$ denote the angle between $z\left(u\right)$ and $W_c$. During training, ArcFace imposes an additive angular margin on the target class:

$${\mathcal{L}}_{\mathrm{Arc}}=-\log {\displaystyle \frac{\exp \left(s\cos ({\theta }_y+m)\right)}{\exp \left(s\cos ({\theta }_y+m)\right)+{\sum }_{c\ne y}\exp \left(s\cos {\theta }_c\right)}}.$$

(2)

Here s is the feature scale and m is the additive angular margin. During inference, the classifier head is discarded. The same normalized embedding space is used for patch-neighbor retrieval, whole-image retrieval, and prototype scoring. This training-inference separation is central to the method: supervision makes the embedding discriminative, while retrieval keeps the prediction inspectable and reference-gallery extensible.

Let ${\mathcal{D}}_{\mathrm{tr}}={\left\{(x_i,y_i)\right\}}_{i=1}^N$ be the labeled reference gallery and x be a query image. The model ranks classes $c\in \{1,\dots ,C\}$ by a score $S\left(c\right|x)$. Unlike a standard classifier, the score is assembled from retrievable local surface neighbors and global prototypes, so the prediction can be inspected and the reference gallery can be updated without retraining the classifier head. This property is important for material surface analysis because new samples can be collected after deployment and inserted into the gallery or prototype bank.

At inference, each evidence branch outputs a normalized class-score vector:

$$\begin{array}{cc}\hfill {\mathbf{q}}_b\left(x\right)& =\left[q_b\left(1\right|x),\dots ,q_b\left(C\right|x)\right],\hfill \\ \hfill {\displaystyle \sum _{c=1}^C}{q}_b\left(c\right|x)& =1,b\in \{s,m,g\},\hfill \end{array}$$

(3)

where s, m, and g denote the single-scale local, multi-scale local, and global prototype branches, respectively. RAGL-MR can, therefore, be viewed as constructing an evidence tuple

$$\mathcal{E}\left(x\right)=\left({\mathbf{q}}_s\left(x\right),{\mathbf{q}}_m\left(x\right),{\mathbf{q}}_g\left(x\right),\mathbf{r}\left(x\right)\right),$$

(4)

where $\mathbf{r}\left(x\right)$ contains reliability cues such as confidence margins, entropy confidence, and branch agreement. The final ranked shortlist is obtained by sorting the fused score:

$$\begin{array}{cc}\hfill \pi \left(x\right)& ={\mathrm{argsort}}_c^{\downarrow }S\left(c\right|x),\hfill \\ \hfill \mathrm{\text{Top-}}k\left(x\right)& =\{{\pi }_1\left(x\right),\dots ,{\pi }_k\left(x\right)\}.\hfill \end{array}$$

(5)

These definitions make the role of the following modules explicit: Section 3.2 defines ${\mathbf{q}}_s$ and ${\mathbf{q}}_m$, Section 3.3 defines ${\mathbf{q}}_g$, and Section 3.4 defines how reliability cues modulate $S\left(c\right|x)$.

3.2. Multi-Scale Local Metric Retrieval for Optical Surface Evidence

Local texture evidence is often decisive, but a fixed patch size assumes that the discriminative pattern appears at a single granularity. This is unreliable for optical material surface images, where fibers, grains, reflections, cracks, and repeated structures can occupy very different spatial extents. A large patch may capture layout context but blur fine texture, whereas a small patch may preserve local surface structure but lose the spatial support needed to distinguish visually similar categories. The proposed local module, therefore, treats scale as part of the retrieval evidence rather than as a fixed preprocessing choice.

For each query image, overlapping surface patches are extracted at multiple sizes and resized to the network input resolution. All patches are embedded by the same metric backbone, so evidence from different scales lies in a shared feature space. This keeps the module simple and avoids training separate networks for each scale. The single-scale branch is retained as an explicit baseline because it tells us whether the additional scale evidence is truly useful.

For a patch p extracted from image x, the backbone $f_{\theta }$ produces a normalized embedding $z=f_{\theta }\left(p\right)$. Each query patch retrieves its K nearest gallery patches. If neighbor j has class $y_j$ and similarity $s_j$, the local score contributed by the patch is

$$q_p\left(c\right)=\sum _{j\in {\mathcal{N}}_K\left(p\right)}\mathbf{1}[y_j=c]{\displaystyle \frac{\exp \left(\alpha s_j\right)}{{\sum }_{\ell \in {\mathcal{N}}_K\left(p\right)}\exp \left(\alpha s_{\ell }\right)}},$$

(6)

where $\alpha$ controls soft-voting sharpness. The image-level local score is the average over all patches:

$$q_{\mathrm{loc}}\left(c\right|x)={\displaystyle \frac{1}{\left|\mathcal{P}\right(x\left)\right|}}\sum _{p\in \mathcal{P}\left(x\right)}{q}_p\left(c\right).$$

(7)

The single-scale branch uses 224-pixel patches, while the multi-scale branch uses 224, 160, and 96-pixel patches with overlapping strides. For scale $r\in \mathcal{R}$, let $q_r\left(c\right|x)$ be the normalized local vote distribution computed from patches at that scale. The final multi-scale local score is

$$q_m\left(c\right|x)=\sum _{r\in \mathcal{R}}{\eta }_r{q}_r\left(c\right|x),\sum _{r\in \mathcal{R}}{\eta }_r=1.$$

(8)

The current evaluation uses uniform scale weights unless validation selection records a different setting. This late aggregation is intentionally conservative: it does not assume that any single scale is always correct, and it allows the Top-k list to accumulate complementary evidence from fine and coarse structures. The resulting branch is more expensive than single-scale retrieval, but it directly targets the shortlist reliability problem.

Figure 2 visualizes exact patch-neighbor evidence for query ID 8. The 224-pixel branch is distracted by structurally similar grooved patches, whereas the 96-pixel and 160-pixel branches retrieve complementary banded neighbors. Aggregating the three scale-level vote distributions promotes the ground-truth class banded to Top-1, providing a concrete example of how receptive-field diversity stabilizes the ranked shortlist.

Figure 2. Exact multi-scale patch-neighbor evidence. The 224-pixel branch is misled by structurally similar grooved patches, whereas the 96-pixel and 160-pixel branches retrieve complementary banded evidence. Aggregating scale-level votes promotes the ground-truth class to Top-1, illustrating how different receptive fields improve ranked local retrieval.

3.3. Multi-Prototype Global Distribution Modeling

Local retrieval can confuse categories that share similar surface fragments. For instance, a small patch of brick, stone, or cracked texture can look similar even when the whole image context differs. Global image evidence can recover these cases, but full-gallery global retrieval is redundant and a single centroid may erase important appearance modes. This motivates MP-GDM, which stores a compact prototype bank for each class.

The key idea is to preserve multiple modes of a material surface category while avoiding the cost of searching all gallery images. For class c, the set of whole-image gallery embeddings ${\mathcal{G}}_c=\{g\left(x_i\right):y_i=c\}$ is clustered by minimizing the within-prototype assignment cost

$$\underset{{\left\{{\mu }_{c,k}\right\}}_{k=1}^{K_c}}{\min }\sum _{g_i\in {\mathcal{G}}_c}\underset{1\le k\le K_c}{\min }{\parallel g_i-{\mu }_{c,k}\parallel }_2^2.$$

(9)

The resulting class prototype bank is

$${\mathcal{P}}_c=\{{\mu }_{c,1},\dots ,{\mu }_{c,K_c}\}.$$

(10)

Given a query whole-image embedding $g\left(x\right)$, its class similarity is aggregated from prototype similarities:

$$r_c\left(x\right)={\mathrm{Agg}}_{k=1}^{K_c}\left(g{\left(x\right)}^{\top }{\mu }_{c,k}\right),$$

(11)

where Agg denotes maximum, mean, or log-mean-exp aggregation. The global score is normalized as

$$q_g\left(c\right|x)={\displaystyle \frac{\exp \left(\tau r_c\left(x\right)\right)}{{\sum }_{c^{\prime }}\exp \left(\tau r_{c^{\prime }}\left(x\right)\right)}}.$$

(12)

The prototype count and aggregation rule are selected on validation data. This is important because datasets differ: DTD categories often benefit from multiple appearance prototypes, while MINC can favor compact global material clusters under the ConvNeXt embedding. MP-GDM, therefore, plays two roles. First, it provides global evidence that complements local surface fragments. Second, it reduces the memory and latency of global retrieval by replacing thousands of reference-gallery vectors with a smaller set of representative prototypes. Unlike standard multi-prototype metric learning that primarily learns class representatives for classification, MP-GDM is used after metric training as a reference-gallery compression and retrieval module: the retained prototypes must preserve ranked shortlist evidence, support gallery maintenance, and expose an explicit accuracy–memory–latency trade-off. The efficiency analysis in Section 4.7 verifies this compression effect.

Algorithm 1 summarizes the validation-selected construction of the MP-GDM prototype bank.

Algorithm 1: MP-GDM prototype-bank construction

1: Input: gallery embeddings and labels ${\left\{(g_i,y_i)\right\}}_{i=1}^N$, candidate counts $\mathcal{K}$, aggregation rules $\mathcal{A}$, validation set $\mathcal{V}$   2: Output: selected prototype bank $\mathcal{P}$ and rule Agg   3: Initialize the best validation score as $-\infty$   4: for each $(K_c,\mathrm{Agg})\in \mathcal{K}\times \mathcal{A}$ do   5:  for each class $c\in \mathcal{C}$ do   6:   Collect ${\mathcal{G}}_c\leftarrow \{g_i:y_i=c\}$   7:   Cluster ${\mathcal{G}}_c$ into $K_c$ groups and normalize the centers   8:   Build ${\mathcal{P}}_c\leftarrow {\left\{{\mu }_{c,k}\right\}}_{k=1}^{K_c}$   9:  end for 10:  Evaluate $q_g$ on $\mathcal{V}$ and update the best configuration 11: end for 12: return$\mathcal{P}$ and Agg

Figure 3 illustrates why one material surface category can require several appearance modes in the shared embedding space. A single centroid may fall between visually distinct modes and match no representative surface well, whereas multiple retained prototypes remain close to the observed sub-clusters. This many-vectors-to-few-prototypes approximation is the central compression mechanism of MP-GDM and supplies compact global evidence for subsequent fusion.

Figure 3. MP-GDM prototype mechanism. Material surface categories may form multi-modal distributions in the embedding space. Point colors denote different classes; open circles denote single centroids, and stars denote retained prototypes. A single centroid can collapse appearance modes, whereas multiple retained prototypes preserve representative modes while replacing redundant gallery vectors. Query-to-prototype aggregation then provides compact global evidence with substantially reduced memory and retrieval latency.

3.4. Validation-Selected Reliability-Aware Fusion

The last module combines three score vectors: single-scale local $q_s\left(c\right|x)$, multi-scale local $q_m\left(c\right|x)$, and global prototype $q_g\left(c\right|x)$. These branches are complementary but not equally reliable for every query. A high-confidence local branch may correctly capture a discriminative surface fragment, while a low-confidence local branch may be dominated by repeated background patterns. Similarly, the global prototype branch can recover image-level material cues but can also be biased by context or illumination. Fusion is, therefore, treated as a validation-selected model-selection problem rather than as a fixed hand-tuned rule.

Figure 4 contrasts three diagnostic outcomes: fusion can fix an incorrect local prediction, preserve a correct and confident prediction, or break a prediction when the available evidence is misleading. The displayed fusion priors are validation-selected static priors rather than per-query learned weights. Accordingly, RAEF is treated as a reliability diagnostic family for interpreting confidence and branch agreement, not as a universally dominant adaptive rule.

Figure 4. Reliability diagnostic cases. The examples show fixed, preserved, and broken predictions under validation-selected static fusion priors. Branch confidence and agreement cues indicate when fusion corrects a local error, preserves a confident prediction, or fails under misleading evidence.

A static validation-selected fusion computes

$$S\left(c\right|x)={\lambda }_s{q}_s\left(c\right|x)+{\lambda }_m{q}_m\left(c\right|x)+{\lambda }_g{q}_g\left(c\right|x),$$

(13)

where ${\lambda }_s+{\lambda }_m+{\lambda }_g=1$. This baseline is deliberately retained because it can be more robust than a hand-designed adaptive rule.

RAEF is evaluated as a query-dependent candidate family. The purpose of RAEF is not to claim that adaptive fusion always outperforms static fusion, but to test whether confidence and disagreement contain useful reliability information. For branch $b\in \{s,m,g\}$, a reliability score ${\rho }_b\left(x\right)$ is derived from margin confidence or entropy confidence:

$${\rho }_{\mathrm{margin}}\left(x\right)=p_{\left(1\right)}\left(x\right)-p_{\left(2\right)}\left(x\right),$$

(14)

$${\rho }_{\mathrm{entropy}}\left(x\right)=1-{\displaystyle \frac{-{\sum }_{c=1}^C{p}_c\left(x\right)\log (p_c\left(x\right)+ϵ)}{\log C}}.$$

(15)

The disagreement-aware candidate additionally adjusts reliability according to branch agreement:

$${\widehat{y}}_b\left(x\right)=\arg \underset{c}{\max }{q}_b\left(c\right|x),b\in \mathcal{B},$$

(16)

$$a_b\left(x\right)={\displaystyle \frac{1}{\left|\mathcal{B}\right|-1}}\sum _{b^{\prime }\ne b}\mathbf{1}\left[{\widehat{y}}_b\left(x\right)={\widehat{y}}_{b^{\prime }}\left(x\right)\right],$$

(17)

$${\tilde{\rho }}_b\left(x\right)=\mathrm{clip}\left({\rho }_b\left(x\right)+{\lambda }_a{a}_b\left(x\right)-{\lambda }_d(1-a_b\left(x\right)),0,1\right),$$

(18)

where $\mathcal{B}=\{s,m,g\}$ and $a_b\left(x\right)$ measures whether branch b’s prediction is supported by the other branches. The branch weights are

$${\omega }_b\left(x\right)={\displaystyle \frac{{\pi }_b{({\tilde{\rho }}_b\left(x\right)+ϵ)}^{\gamma }}{{\sum }_{b^{\prime }}{\pi }_{b^{\prime }}{({\tilde{\rho }}_{b^{\prime }}\left(x\right)+ϵ)}^{\gamma }}},$$

(19)

and the adaptive score is

$$S_{\mathrm{RAEF}}\left(c\right|x)=\sum _{b\in \{s,m,g\}}{\omega }_b\left(x\right)q_b\left(c\right|x).$$

(20)

Static fusion, confidence-only RAEF, and disagreement-aware RAEF are all selected only on validation data. In the current results, the best selected form is dataset-dependent, so RAEF is claimed as a reliability-aware candidate family and diagnostic tool rather than as a universally dominant fusion rule. This differs from conventional ensemble weighting in two ways: the fused branches correspond to inspectable evidence sources rather than independent classifiers, and the reliability variables are reported as diagnostic quantities through confidence-bin, disagreement, calibration, and fixed/broken analyses. This conservative positioning is essential: the experiments show that reliability cues help identify low-confidence and branch-disagreement samples, but the final fusion form should still be chosen by validation rather than assumed in advance.

In summary, the method forms a retrieval pipeline rather than a single classifier. The local branch exposes patch-level optical surface evidence, the multi-scale branch broadens the receptive-field evidence, MP-GDM supplies compact global support, and validation-selected fusion converts these branches into an evidence-aware Top-k shortlist. This division of labor is the basis for the experiments: each quantitative block tests one part of the pipeline and then checks whether the combined system improves ranked retrieval reliability.

4. Experimental Results and Analysis

4.1. Experimental Setting

The experiments are organized around the three proposed modules: multi-scale local retrieval, MP-GDM global evidence, and reliability-aware fusion diagnostics. Instead of presenting isolated numbers, each experiment is designed to answer one question. The overall comparison tests whether global-local fusion improves retrieval reliability. The ablation study separates the effects of local scale, global prototypes, and fusion choice. The reliability analysis asks where the gains come from. The external-dataset and runtime analyses examine whether the conclusion is still meaningful beyond the main DTD/MINC setting.

DTD has 47 texture classes with official train, validation, and test splits. The DTD overall-comparison rows report mean and sample standard deviation over the first five official splits, while detailed ablations and reliability diagnostics use split 1 unless otherwise specified. MINC-2500 has 23 material classes with 2500 images per class and five official train/test splits. The ConvNeXt MINC comparison reports mean and sample standard deviation over the five official splits; within each split, a stratified holdout from the training set is used for checkpoint and fusion selection. FMD is split deterministically per class into 50/25/25 train/validation/test images. KTH-TIPS2-b is split by physical sample identity, using samples a/b for training, sample c for validation, and sample d for testing. These public datasets are used as texture/material surface image benchmarks for evaluating scale variation, appearance diversity, and ambiguous local evidence; they are not presented as controlled photonic-acquisition datasets.

All headline rows are validation-selected. Any fixed fusion weight selected directly on the test set is treated only as a diagnostic upper bound and is not used as a main claim. We report Top-1, Top-5, Top-10, and MRR. Top-1 measures the first prediction, while Top-5 and Top-10 measure whether the correct class remains in a useful retrieval shortlist. This distinction is central to the paper because the target use case is evidence-aware optical material surface retrieval, human verification, and reference-gallery maintenance rather than only closed-set classification.

The main backbone is ConvNeXt-Tiny with a 512-dimensional ArcFace embedding. Multi-scale local retrieval uses patch sizes 224, 160, and 96. MP-GDM sweeps prototype count and aggregation mode on validation data. RAEF sweeps branch priors, confidence type, and reliability sharpness. To contextualize the retrieval-oriented claim, we additionally report a matched ConvNeXt-Tiny closed-set classifier, CLIP frozen image-level kNN, DINOv2 frozen image-level kNN, and earlier DINO/ViT metric controls. All local-patch runtime measurements are recorded on an RTX 4090 server. Unless otherwise specified, reported fusion results are selected once on validation data and then evaluated once on the held-out test set.

The experiments were implemented in Python 3.12 using PyTorch 2.5.1+cu124 with CUDA-enabled GPU execution. The main software dependencies included timm 1.0.27 for model backbones, scikit-learn 1.7.2 for clustering and metric utilities, SciPy 1.15.3, pandas 2.3.3, NumPy 2.2.6, Pillow 12.2.0, seaborn 0.13.2, and matplotlib 3.10.9 for analysis and visualization. Runtime measurements were obtained on an NVIDIA GeForce RTX 4090 server with CUDA 12.4; exact scripts, configuration files, selected result files, and environment notes are provided in the accompanying GitHub repository.

Table 1 and Table 2 summarize the fixed retrieval settings and validation-selected prototype/fusion configurations used throughout the reported experiments.

Table 1. Retrieval configuration used for validation-selected experiments. Patch settings and local voting parameters are fixed by the evaluation protocol.

Dataset	Protocol	Backbone	Patch Sizes/Strides	K	Voting Temperature $\mathit{\alpha }$
DTD	split-1 diagnostics/first five official splits	ConvNeXt-Tiny ArcFace	224/160/96, 112/80/48	15	20
MINC	five official splits	ConvNeXt-Tiny ArcFace	224/160/96, 112/80/48	15	20
FMD	fixed train/validation/test	ViT-B/DINO ArcFace	224/160/96, 112/80/48	15	20
KTH	sample-wise split	ViT-B/DINO ArcFace	224/160/96, 112/80/48	15	20

Table 2. Validation-selected prototype and fusion configurations. Prototype and fusion settings are selected on validation data or recorded per split in the corresponding JSON result files.

Dataset	Protocol	Prototype Setting	Aggregation	Fusion Type and Priors	Selected Metric
DTD	split-1 diagnostic	$K_c=5$	log-mean-exp	static, (0.25, 0.50, 0.25), no confidence cue	Top-1
DTD	first five official splits	per-split $K_c\in \{1,3,5,10\}$	max/log-mean-exp	per-split static or RAEF, stored in split JSON	Top-1
MINC	five official splits	per-split $K_c\in \{1,2,3,5,10\}$	max/log-mean-exp/weighted LSE	per-split selected; split 1 uses (0.10, 0.00, 0.90)	Top-5
FMD	fixed train/validation/test	$K_c=5$	max	DRAEF, (0.00, 0.80, 0.20), entropy, $\gamma =0.5$	Top-1
KTH	sample-wise split	$K_c=1$	max	DRAEF, (0.00, 0.10, 0.90), margin, $\gamma =1.0$	Top-1

4.2. Quantitative Comparison Across Optical Surface Image Benchmarks

This experiment answers whether the complete global-local retrieval framework improves ranked optical material surface recognition over its individual evidence branches. Table 3 summarizes the overall comparison and directly evaluates the overall framework. DTD rows report mean and sample standard deviation over the first five official splits. The MINC rows report five-split mean and standard deviation. The comparison includes global retrieval, single-scale local retrieval, multi-scale local retrieval, MP-GDM, and final fusion, so the reader can see how the method moves from individual evidence branches to the final ranked shortlist.

Table 3. Overall comparison under the common ConvNeXt-Tiny metric backbone. Values are percentages except MRR. DTD and MINC rows report mean ± sample standard deviation over five official splits.

Dataset	Method	Top-1	Top-5	Top-10	MRR	Note
DTD	Global image retrieval	73.29 ± 0.30	81.65 ± 0.92	83.36 ± 0.97	0.7758 ± 0.0053	5-split
DTD	Single-scale local retrieval	73.89 ± 0.48	85.26 ± 1.16	86.74 ± 1.06	0.7936 ± 0.0074	local
DTD	Multi-scale local retrieval	74.05 ± 0.19	91.43 ± 0.70	94.15 ± 0.57	0.8167 ± 0.0017	multi-scale
DTD	MP-GDM only	73.15 ± 0.40	90.74 ± 0.43	94.60 ± 0.31	0.8109 ± 0.0020	prototype
DTD	Final MP-GDM fusion	74.06 ± 0.11	92.61 ± 0.61	96.11 ± 0.23	0.8221 ± 0.0015	val-selected
MINC	Global image retrieval	86.00 ± 0.32	92.51 ± 0.30	94.38 ± 0.43	0.8947 ± 0.0023	5-split
MINC	Single-scale local retrieval	84.50 ± 0.29	95.59 ± 0.30	96.77 ± 0.28	0.8973 ± 0.0027	local
MINC	Multi-scale local retrieval	79.75 ± 0.38	96.59 ± 0.39	98.76 ± 0.30	0.8717 ± 0.0024	multi-scale
MINC	MP-GDM only	85.70 ± 0.25	97.47 ± 0.34	99.03 ± 0.23	0.9087 ± 0.0022	prototype
MINC	Final MP-GDM fusion	85.84 ± 0.25	97.63 ± 0.28	99.15 ± 0.10	0.9105 ± 0.0016	val-selected

The key result is not universal Top-1 or Top-k dominance: the global branch is already strong in Top-1 on MINC, and later matched controls show that closed-set classifiers and DINOv2 features can be stronger on some image-level metrics. The intended use case is different. An optical material surface retrieval system should keep the correct category in the candidate list while also exposing the evidence behind that ranking, so shortlist quality, local surface evidence, prototype compression, and reliability diagnostics are evaluated together.

On DTD, multi-scale local retrieval raises five-split Top-10 from 86.74 ± 1.06% to 94.15 ± 0.57%, and the final MP-GDM fusion reaches 96.11 ± 0.23% Top-10. This supports the first contribution: scale-expanded local evidence captures additional surface fragments that a single patch size misses. On MINC, final fusion improves over single-scale local retrieval by 2.04 points in Top-5, 2.38 points in Top-10, and 0.0132 MRR. This supports the second and third contributions: compact global prototypes and validation-selected fusion improve the ranked list even when Top-1 is already competitive.

Table 4 and Table 5 provide the split-wise stability behind these headline rows. They are placed as compact companion tables to keep the discussion of five-split behavior close to the overall comparison.

Table 4. DTD split-wise final-fusion stability over the first five official splits. Values are percentages except MRR.

Split	Top-1	Top-5	Top-10	MRR
1	74.15	92.23	95.74	0.8217
2	74.10	92.50	96.17	0.8214
3	73.88	91.97	96.12	0.8202
4	74.04	92.77	96.12	0.8234
5	74.15	93.56	96.38	0.8240
Mean ± std	74.06  ±0.11	92.61  ±0.61	96.11  ±0.23	0.8221  ±0.0015

Table 5. MINC-2500 split-wise final-fusion stability. Values are percentages except MRR.

Split	Top-1	Top-5	Top-10	MRR
1	85.91	97.57	99.08	0.9103
2	85.51	97.46	99.15	0.9087
3	86.21	97.76	99.13	0.9129
4	85.83	97.32	99.06	0.9094
5	85.76	98.03	99.32	0.9110
Mean ± std	85.84  ±0.25	97.63  ±0.28	99.15  ±0.10	0.9105  ±0.0016

Table 4 shows that the DTD headline is also stable across multiple official splits. Top-1 varies by only 0.27 points, while Top-10 remains above 95.7% for all five splits. This reduces the risk that the DTD result is a favorable split-1 artifact and makes the retrieval-oriented claim more suitable for a journal submission.

Table 5 shows that the MINC result is not driven by a single favorable split. Top-5 remains between 97.32% and 98.03%, and Top-10 remains between 99.06% and 99.32%. The small standard deviation is important for a journal submission because it demonstrates that the reported MINC improvement is not a split-1 artifact. The stability of these shortlist metrics is the main evidence for the retrieval-focused claim.

4.3. Ablation Study

This experiment answers which proposed component is responsible for the final behavior: multi-scale local retrieval, MP-GDM global evidence, or validation-selected fusion. Table 6 isolates the role of each branch and fusion family on split-1 diagnostic runs. This experiment is deliberately more detailed than the main table because it explains how the final method is constructed. The branch baselines are test results. Fusion-family rows marked “val-best” are validation metrics, and the selected row is evaluated once on the test set. This organization separates method selection from test reporting and avoids presenting test-set tuning as a headline result.

Table 6. Ablation study with validation-selected fusion. Test branch baselines are split-1 diagnostic test results. Validation-only rows are used for model selection and are not test-set performance; selected rows are evaluated once on the held-out test set.

Dataset	Method	Top-1	Top-5	Top-10	MRR	Note
Test branch baselines
DTD	Global image retrieval	73.35	80.85	82.66	0.7732	test baseline
DTD	Single-scale local retrieval	73.51	83.99	85.69	0.7869	test baseline
DTD	Multi-scale local retrieval	74.04	91.06	94.04	0.8162	test baseline
DTD	Best MP-GDM only	73.51	90.64	94.04	0.8118	test baseline
Validation-only fusion candidates
DTD	Static MP-GDM fusion, val-best	74.36	92.02	95.21	–	validation only
DTD	Confidence-only RAEF, val-best	74.20	92.02	95.21	–	validation only
DTD	Disagreement-aware RAEF, val-best	74.31	91.97	95.43	–	validation only
Selected test model
DTD	Static MP-GDM fusion, selected	74.15	92.23	95.74	0.8217	test
Test branch baselines
MINC	Global image retrieval	86.23	92.31	94.10	0.8956	split-1 diagnostic
MINC	Single-scale local retrieval	85.25	95.15	96.43	0.9000	split-1 diagnostic
MINC	Multi-scale local retrieval	79.58	96.02	98.45	0.8693	split-1 diagnostic
MINC	Best MP-GDM only	86.12	96.85	98.50	0.9103	split-1 diagnostic
Validation-only fusion candidates
MINC	Static MP-GDM fusion, val-best	86.17	97.46	98.80	–	validation only
MINC	Confidence-only RAEF, val-best	86.14	97.48	98.80	–	validation only
MINC	Disagreement-aware RAEF, val-best	86.17	97.46	98.80	–	validation only
Selected test model
MINC	Static MP-GDM fusion, selected	86.23	97.15	98.66	0.9108	test

The ablation confirms the three-module logic. Multi-scale local retrieval provides the largest shortlist gain on DTD, which is consistent with DTD’s fine-grained texture patterns. MP-GDM gives compact global evidence and strong MRR on MINC, where material surface categories often benefit from whole-image distribution cues. Static MP-GDM fusion is selected over hand-designed disagreement-aware RAEF in the compact grid, so adaptive fusion is interpreted conservatively: branch disagreement is useful for diagnosis and future routing, but it is not yet a consistently superior final rule. This is why the final method is named MP-GDM fusion rather than overstating DRAEF as the universal best model.

Figure 5 visualizes the corresponding Top-k ablation curves and shows that the main improvement is concentrated in ranked shortlist reliability rather than only in Top-1 accuracy.

Figure 5. Top-k curves for the validation-selected ablation: (a) DTD and (b) MINC. The main gain appears beyond Top-1, indicating stronger shortlist reliability.

4.4. Extended Baselines and Cross-Dataset Checks

This experiment answers whether the observed behavior is only a byproduct of using a modern backbone, and whether the retrieval design remains meaningful when compared with closed-set and foundation-feature controls. Table 7 reports three matched controls. The ConvNeXt-Tiny closed-set classifier tests the same backbone family under an ordinary supervised classifier head. CLIP and DINOv2 frozen image-level kNN test off-the-shelf foundation features under an expandable retrieval protocol, but without local patch evidence, MP-GDM prototypes, or reliability fusion. These controls are not intended to replace a comprehensive closed-set SOTA comparison; they clarify the relationship between retrieval-oriented reliability, foundation-feature transfer, and closed-set recognition accuracy for optical material surface analysis.

Table 7. Matched closed-set and foundation-feature controls. Values are percentages and are reported as mean ± sample standard deviation over the same five-split protocols used for the DTD and MINC headline results. The controls clarify whether ordinary closed-set classification or frozen foundation features are sufficient under comparable evaluation settings.

Dataset	Feature/Backbone	Method	Train.	Top-1	Top-5	Top-10	Interpretation
DTD	ConvNeXt-Tiny	Closed-set classifier	CE	72.11 ± 1.14	93.39 ± 0.80	97.15 ± 0.34	closed-set, no evidence
DTD	CLIP	Frozen image kNN	none	69.26 ± 1.31	90.61 ± 0.61	95.19 ± 0.45	foundation retrieval
DTD	DINOv2	Frozen image kNN	none	77.36 ± 0.87	94.93 ± 0.66	97.68 ± 0.34	strong foundation
DTD	ConvNeXt-Tiny ArcFace	Final MP-GDM fusion	metric	74.06 ± 0.11	92.61 ± 0.61	96.11 ± 0.23	evidence-aware retrieval
MINC	ConvNeXt-Tiny	Closed-set classifier	CE	85.12 ± 0.26	98.39 ± 0.12	99.63 ± 0.07	closed-set shortlist
MINC	CLIP	Frozen image kNN	none	66.60 ± 0.35	92.63 ± 0.41	97.02 ± 0.27	foundation retrieval
MINC	DINOv2	Frozen image kNN	none	69.87 ± 0.82	94.85 ± 0.58	98.16 ± 0.25	foundation retrieval
MINC	ConvNeXt-Tiny ArcFace	Final MP-GDM fusion	metric	85.84 ± 0.25	97.63 ± 0.28	99.15 ± 0.10	prototype retrieval

Figure 6 further compares the Top-k behavior of the matched closed-set and foundation-feature controls in Table 7.

Figure 6. Top-k curves for extended backbone diagnostics: (a) DTD and (b) MINC. The curves complement Table 7 by showing how shortlist behavior changes across retrieval controls.

The extended controls sharpen the scope of the claim. DINOv2 frozen image kNN is particularly strong on DTD, showing that large self-supervised representations can provide excellent image-level retrieval without task-specific training. The proposed framework is, therefore, not claimed as a universal Top-1 or Top-k replacement for foundation features. Its contribution lies in evidence-preserving optical surface retrieval: it exposes patch-neighbor evidence, supports prototype-compressed global retrieval, provides branch-level reliability diagnostics, and keeps the reference gallery extensible under a metric protocol.

On MINC, the closed-set ConvNeXt classifier gives the strongest Top-5 and Top-10 among the matched controls, whereas RAGL-MR retains competitive Top-1 and provides retrieval evidence and prototype compression that ordinary classifiers do not expose. Overall, Table 7 changes the interpretation from pure accuracy dominance to retrieval-oriented complementarity. Closed-set classifiers and frozen foundation features remain strong baselines, especially for image-level ranking. RAGL-MR is designed for a different operating condition: the reference gallery may need to be inspected or expanded, local surface evidence should be visible, and the system should report reliability behavior rather than only a single label. This is why the following reliability and efficiency analyses are central to the paper rather than auxiliary results.

The large-grid diagnostic also shows why the RAEF claim must be cautious. DTD can select a disagreement-aware candidate in the larger grid, but MINC still selects confidence-only RAEF under split-1 diagnostics. This dataset dependence supports the use of validation-selected fusion families rather than a fixed adaptive rule. It also motivates a future learned gating module, where reliability cues can be trained rather than hand combined.

Table 8 extends the evaluation to FMD and KTH-TIPS2-b. This experiment is used as cross-dataset validation rather than as a large hyperparameter search. FMD shows a positive trend: ArcFace alignment, multi-scale local evidence, and validation-selected DRAEF each add useful retrieval evidence. KTH is more mixed: multi-scale retrieval strongly improves Top-5 and Top-10, but the validation-selected DRAEF candidate does not improve Top-1 over multi-scale local retrieval. This negative result is retained because it exposes the stability limits of hand-designed adaptive fusion and prevents the paper from overstating reliability-aware fusion.

Table 8. External material-dataset evaluation on FMD and KTH-TIPS2-b with the ViT-B/DINO backbone. Values are percentages; KTH denotes KTH-TIPS2-b.

Dataset	Method	Top-1	Top-5	Top-10	Note
FMD	Frozen image kNN	65.20	89.60	95.20	external baseline
FMD	ArcFace global score	76.40	87.60	100.00	whole-image score
FMD	Single-scale local retrieval	78.40	90.80	100.00	224 patches
FMD	Multi-scale local retrieval	79.60	97.20	100.00	224/160/96
FMD	Selected DRAEF candidate	80.00	98.00	100.00	val-selected test
KTH	Frozen image kNN	84.93	93.69	96.04	external baseline
KTH	ArcFace global score	78.87	86.87	96.13	whole-image score
KTH	Single-scale local retrieval	79.29	87.63	96.13	224 patches
KTH	Multi-scale local retrieval	79.21	94.87	99.75	224/160/96
KTH	Selected DRAEF candidate	78.96	95.37	98.91	val-selected test

4.5. Robustness, Calibration, Large-Gallery Scaling, and Industrial Surface Check

This subsection summarizes additional revision analyses requested by the reviewers. The goal is not to redefine the main benchmark protocol, but to test whether the retrieval interpretation remains reasonable under simple acquisition-like perturbations, large-gallery scaling, reliability calibration, and a real industrial surface-defect proxy. Calibration and large-gallery scaling are further visualized in Section 4.6 and Section 4.7; here we first report the perturbation and industrial image-folder checks.

Table 9 reports a DTD robustness stress test over the first five official splits, and Figure 7 visualizes the corresponding Top-5 robustness trends. Brightness reduction and Gaussian noise cause only moderate changes, while Gaussian blur is clearly more damaging. This behavior is consistent with optical surface analysis: blur removes high-frequency local texture evidence that patch-neighbor retrieval needs. DINOv2 remains the strongest frozen image-level retrieval control on DTD, so these results should not be interpreted as universal dominance of RAGL-MR over foundation features. Instead, they show that the proposed retrieval pipeline maintains useful Top-5 shortlist behavior under simple perturbations while preserving inspectable evidence.

Table 9. DTD robustness stress test over the first five official splits. Values are mean ± sample standard deviation; accuracy values are percentages. DINOv2 and CLIP are frozen image-level retrieval controls with clean galleries and perturbed queries.

Perturbation	Method	Top-1	Top-5	MRR
Clean	Final RAGL-MR	73.53 ± 0.45	91.19 ± 0.86	0.8136 ± 0.0041
	Single-scale local	74.04 ± 0.19	91.23 ± 0.92	0.8158 ± 0.0029
	MP-GDM/global prototype	73.29 ± 0.30	81.65 ± 0.92	0.7758 ± 0.0053
	DINOv2 frozen image kNN	77.36 ± 0.87	94.93 ± 0.66	0.8507 ± 0.0069
	CLIP frozen image kNN	69.26 ± 1.31	90.61 ± 0.61	0.7886 ± 0.0076
Brightness 0.70	Final RAGL-MR	73.26 ± 0.62	91.23 ± 0.94	0.8118 ± 0.0030
	Single-scale local	73.78 ± 0.34	91.14 ± 0.96	0.8140 ± 0.0048
	MP-GDM/global prototype	72.99 ± 0.54	81.35 ± 0.76	0.7729 ± 0.0055
	DINOv2 frozen image kNN	77.49 ± 0.85	94.89 ± 0.68	0.8512 ± 0.0068
	CLIP frozen image kNN	68.80 ± 1.16	90.64 ± 0.40	0.7854 ± 0.0066
Gaussian noise 0.05	Final RAGL-MR	72.87 ± 0.41	90.67 ± 0.66	0.8079 ± 0.0031
	Single-scale local	73.06 ± 0.23	90.49 ± 0.77	0.8082 ± 0.0029
	MP-GDM/global prototype	72.24 ± 0.66	81.44 ± 0.73	0.7685 ± 0.0045
	DINOv2 frozen image kNN	76.67 ± 0.66	94.74 ± 0.43	0.8465 ± 0.0056
	CLIP frozen image kNN	68.66 ± 1.31	90.46 ± 0.48	0.7845 ± 0.0071
Gaussian blur 1.50	Final RAGL-MR	70.05 ± 0.38	88.82 ± 0.91	0.7837 ± 0.0052
	Single-scale local	69.36 ± 0.74	88.64 ± 0.92	0.7786 ± 0.0070
	MP-GDM/global prototype	70.05 ± 0.32	79.55 ± 0.72	0.7490 ± 0.0040
	DINOv2 frozen image kNN	76.35 ± 0.61	94.39 ± 0.65	0.8430 ± 0.0061
	CLIP frozen image kNN	66.69 ± 1.29	89.01 ± 0.59	0.7665 ± 0.0073

Figure 7. DTD five-split perturbation stress test. The analysis compares clean images, brightness scaling, Gaussian noise, and Gaussian blur under the same retrieval evaluator. The result is reported as a robustness diagnostic rather than as a new controlled photonic-acquisition benchmark.

To better connect the retrieval protocol with industrial optical inspection scenarios, we further conducted a clean image-folder retrieval sanity check on the Magnetic Tile Surface Defect Dataset. The original repository contains paired surface images and pixel-level label files. We audited the prepared image folder and excluded all .png mask/label files, retaining only 1344 .jpg surface images. The final stratified split uses seed 20260705 with 806 training images, 268 validation images, and 270 test images. The six classes are strongly imbalanced, with class counts 115/85/57/32/952/103 for MT_Blowhole, MT_Break, MT_Crack, MT_Fray, MT_Free, and MT_Uneven, respectively. This experiment is an industrial surface-defect retrieval sanity check; it does not use segmentation masks, does not train a new model, and is not presented as a controlled optical-acquisition study.

Table 10 should be interpreted cautiously, and Figure 8 provides the associated Top-k retrieval curves. ConvNeXt-Tiny kNN obtains the highest overall Top-1, largely because the defect-free class is dominant. MP-GDM does not universally improve every metric, but it raises Macro Top-1 for ConvNeXt-Tiny from 56.01% to 70.65% and for DINOv2 from 19.61% to 62.57%, indicating better minority-class behavior under prototype-compressed retrieval. This supports the reviewer-response claim that prototype-compressed retrieval can complement strong image embeddings in industrial surface-defect retrieval, without claiming that RAGL-MR universally outperforms CLIP, DINOv2, or ConvNeXt.

Table 10. Real industrial surface-defect image-folder retrieval sanity check on the Magnetic Tile dataset. Macro Top-1 is reported because the dataset is strongly class-imbalanced. Values are percentages except MRR.

Method	Top-1	Macro Top-1	Top-3	MRR
ConvNeXt-Tiny frozen image kNN	83.33	56.01	99.26	0.9130
ConvNeXt-Tiny MP-GDM prototype retrieval	75.93	70.65	97.78	0.8656
CLIP frozen image kNN	71.11	19.05	99.26	0.8461
CLIP MP-GDM prototype retrieval	75.19	51.96	96.67	0.8558
DINOv2 frozen image kNN	71.85	19.61	98.89	0.8526
DINOv2 MP-GDM prototype retrieval	76.30	62.57	96.30	0.8652

Figure 8. Top-k behavior on the clean Magnetic Tile image-folder sanity check. Top-5 is nearly saturated because the dataset has only six classes; the main interpretation, therefore, relies on Top-1, Macro Top-1, Top-3, MRR, and per-class diagnostics.

4.6. Reliability Analysis

Reliability analysis tests the third contribution: whether fusion helps in cases where local retrieval is uncertain, or branches disagree. This experiment answers where the final fusion gains come from, rather than only reporting the average Top-1 change. Table 11 groups samples by local-branch confidence. The largest gains appear in low-confidence samples, while high-confidence predictions are preserved.

Table 11. Confidence-bin analysis versus the single-scale local baseline. Bins are formed by local-branch confidence.

Dataset	Bin	Count	Local Top-1	Final Top-1	Delta
DTD	low	627	49.76	51.99	2.23
DTD	mid	627	84.53	84.69	0.16
DTD	high	626	86.42	86.42	0.00
MINC	low	1917	64.42	67.19	2.76
MINC	mid	1917	93.74	93.79	0.05
MINC	high	1916	97.55	97.55	0.00

Branch disagreement provides a second diagnostic axis. If all branches predict the same class, fusion has little room to change the decision. If branches disagree, the query is more ambiguous and complementary evidence may become more valuable. Table 12 shows that branch disagreement is useful but dataset-dependent: MINC all-disagree samples show strong recovery, whereas the small DTD all-disagree subset remains unstable. Figure 9 visualizes both the confidence-bin and branch-disagreement diagnostics. Therefore, disagreement should be interpreted as an ambiguity cue rather than a guarantee that fusion will improve every sample.

Table 12. Branch-disagreement analysis versus single-scale local retrieval.

Dataset	Agreement Bin	Count	Base Top-1	Final Top-1	Fixed	Broken
DTD	all disagree	15	26.67	20.00	2	3
DTD	one pair agrees	154	26.62	34.42	20	8
DTD	all agree	1711	78.20	78.20	0	0
MINC	all disagree	86	16.28	51.16	41	11
MINC	one pair agrees	749	60.61	64.22	80	53
MINC	all agree	4915	90.19	90.19	0	0

Figure 9. Reliability diagnostic plots. (a) Confidence-bin analysis shows that the final model primarily improves low-confidence samples while preserving high-confidence predictions. (b) Branch-disagreement diagnostics show that disagreement indicates ambiguous samples where complementary evidence may help, especially on MINC all-disagree cases, but the small DTD all-disagree subset remains unstable.

Paired correctness gives a stricter view than aggregate accuracy. A method can improve average accuracy while still breaking many previously correct samples. Table 13 compares sample-level fixes and breaks relative to the local baseline. Figure 10 complements the paired table with bootstrap confidence intervals for selected reliability subsets. The final model fixes more images than it breaks on both datasets, and the McNemar tests indicate that the paired changes are not random under these diagnostic splits.

Table 13. Paired Top-1 analysis versus single-scale local retrieval. Delta and CI95 are percentage points.

Dataset	Base Correct	Final Correct	Fixed	Broken	Delta	CI95	McNemar p
DTD	1383	1398	30	15	0.80	[0.11, 1.49]	0.03689
MINC	4901	4955	119	65	0.94	[0.49, 1.41]	9.337 × 10−5

Figure 10. Bootstrap confidence intervals for Top-1 delta relative to the local baseline on selected reliability subsets. Green and red markers denote positive and negative Top-1 deltas, respectively.

To further examine the reliability interpretation, Figure 11 reports confidence–accuracy calibration curves and expected calibration error (ECE) for the local branch, global image retrieval branch, and final fusion. The calibration analysis does not turn RAEF into a full probabilistic uncertainty model; rather, it tests whether confidence-derived cues are aligned with empirical correctness. On DTD, the final fusion reduces ECE from 0.2002 for the single-scale local branch and 0.2296 for the global branch to 0.1579. On MINC, the final fusion substantially improves Top-k retrieval while remaining between the local and global branches in ECE. These results support the use of margin, entropy, and agreement as practical reliability proxies, while also confirming that validation selection remains necessary.

Figure 11. Calibration analysis for reliability-aware fusion. The curves compare confidence and empirical accuracy for the single-scale local branch, global image retrieval branch, and final RAGL-MR fusion. ECE values are computed with 10 confidence bins.

4.7. Efficiency and Failure Analysis

This experiment answers two practical questions: whether MP-GDM has real efficiency value, and where the current system still fails. Table 14 reports prototype-compressed global retrieval and verifies the second contribution from a computational perspective. On MINC, reducing the gallery from 48,875 vectors to 506 prototypes cuts memory from 95.46 MB to 0.99 MB and mean query latency from 0.505 ms to 0.047 ms, while Top-10 improves from 91.46% to 93.43%. Thus, MP-GDM provides compact global support rather than an expensive full-gallery search.

Table 14. Prototype-compressed global retrieval efficiency on the RTX 4090 environment. Latency is mean ms/query over repeated runs.

Dataset	Setting	Vectors	Memory MB	Compression	Latency ms/Query	Top-1	Top-5	Top-10
DTD	Full gallery	1880	3.67	1.0×	0.058	73.40	76.06	77.18
DTD	Prototype, 20×	94	0.18	20.0×	0.038	73.09	87.18	91.60
DTD	Prototype, 40×	47	0.09	40.0×	0.049	72.82	89.36	93.03
MINC	Full gallery	48,875	95.46	1.0×	0.505	84.89	90.45	91.46
MINC	Prototype, 20×	2461	4.81	19.9×	0.052	84.63	92.17	93.44
MINC	Prototype, 96.6×	506	0.99	96.6×	0.047	84.45	92.09	93.43

Figure 12 provides a feature-space view of the MP-GDM mechanism. The open circles represent single centroids and the stars represent retained prototypes. The visualization shows why MP-GDM is not merely a standard single-prototype classifier: material classes occupy multi-modal regions in the metric space, and a single centroid can fall between appearance modes. Retaining several normalized prototypes per class preserves representative modes while still replacing many redundant gallery vectors.

Figure 12. Feature-space visualization of MP-GDM on selected MINC gallery embeddings. Open circles denote single class centroids, whereas stars denote retained multi-prototypes.

Figure 13 summarizes the accuracy–memory–latency trade-off as prototype-compression Pareto plots for DTD and MINC.

Figure 13. Prototype-compression Pareto plots for (a) DTD and (b) MINC. Marker color denotes latency per query. Prototype compression reduces memory and latency while preserving global retrieval accuracy.

The measured efficiency results also permit a large-gallery stress analysis. Figure 14 extrapolates exact full-gallery retrieval under redundant gallery expansion and compares it with the retained MP-GDM prototype bank. This is a computational scaling analysis rather than a new-category experiment: exact retrieval is extrapolated linearly from measured memory and latency, whereas MP-GDM keeps the selected prototype bank fixed when redundant reference samples are added. The gap illustrates the operational value of prototype compression for optical inspection systems whose reference galleries may grow after deployment.

Figure 14. Large-gallery scaling stress analysis. Full-gallery exact retrieval is extrapolated from measured latency and memory as the reference gallery grows, while MP-GDM keeps a compact retained prototype bank.

Table 15 reports full local-patch evaluation time. The multi-scale branch is substantially more expensive because it extracts and embeds many more patches per query. The current evaluator processes query images sequentially, so GPU memory can be underused during patch preparation and score-cache writing. This affects runtime but not reported accuracy, and it motivates a parallel query scheduler.

Table 15. Local-patch metric retrieval runtime on the RTX 4090 environment. Time includes image loading, patch extraction, embedding, neighbor scoring, and score-cache writing. Accuracy columns report the local branch from the same run.

Dataset	Setting	Images	Avg. Patches/ Image	Total Time (s)	Seconds/ Image	Top-1	Top-5	Top-10
DTD	Single-scale local	1880	4.68	170.41	0.091	73.51	83.99	85.69
DTD	Multi-scale local	1880	44.89	493.18	0.262	74.04	91.06	94.04
MINC	Single-scale local	5750	4.00	786.90	0.137	85.25	95.15	96.43
MINC	Multi-scale local	5750	38.00	2279.27	0.396	79.58	96.02	98.45
FMD	Single-scale local	250	6.00	24.76	0.099	78.40	90.80	100.00
FMD	Multi-scale local	250	53.00	66.47	0.266	79.60	97.20	100.00
KTH	Single-scale local	1188	4.12	104.03	0.088	79.29	87.63	96.13
KTH	Multi-scale local	1188	39.23	351.66	0.296	79.21	94.87	99.75

Failure-case analysis connects the quantitative results to category behavior. Figure 15 shows that the model improves DTD classes such as porous and veined, and MINC classes such as paper and skin. Regressions on fabric, carpet, and mirror show that global context and local neighbors can still mislead the current fusion rule. These cases motivate learned branch calibration and stronger multi-split diagnostics.

Figure 15. Failure-case class analysis for (a) DTD and (b) MINC. Green bars indicate classes improved by the final fusion, whereas red bars indicate classes with regressions. The method improves several low-confidence or structure-sensitive classes but still has category-specific regressions.

5. Discussion

The results should be interpreted in relation to three groups of published work. First, modern closed-set and foundation-feature methods provide very strong image-level representations. ConvNeXt, ViT-family backbones, CLIP, DINO, and DINOv2 improve global image recognition and nearest-neighbor structure [13,14,15,16,17]. Our extended controls confirm this strength: DINOv2 frozen retrieval is particularly strong on DTD, and a closed-set ConvNeXt classifier remains highly competitive on MINC. Therefore, the contribution of RAGL-MR should not be read as a universal replacement for foundation features or closed-set classifiers. Its value is instead complementary: it organizes a metric embedding into local patch-neighbor evidence, prototype-compressed global retrieval, and reliability diagnostics, which are useful when optical surface recognition requires inspectable evidence and an updateable reference gallery.

Second, classical multi-scale structural representations such as ridgelets, curvelets, and phase congruency remain important references for optical image analysis [30,31,32,33,34]. They are designed to capture line-, curve-, edge-, and phase-consistent structures across scales, and they are valuable for denoising, feature extraction, and handcrafted structural analysis. RAGL-MR does not replace these methods as general signal-processing tools. Rather, it addresses a different retrieval setting: a user has a labeled material or surface-defect reference gallery and needs a ranked shortlist, patch-neighbor evidence, and compact gallery maintenance. This explains why the proposed method uses metric embeddings and retrieved neighbors rather than handcrafted transforms as the main representation, while still preserving the core insight that multi-scale structure matters for optical surface interpretation.

Third, recent texture/material recognition work improves feature aggregation, material resources, local/global primitive modeling, and cross-scale relations [27,28,29]. These studies motivate stronger representations, but they are usually evaluated as recognition or representation-learning methods. RAGL-MR is narrower and more system-oriented: it asks whether local evidence, global prototypes, and reliability cues can be combined into an evidence-aware retrieval workflow. The Magnetic Tile sanity check further supports this system interpretation. Although it is not a controlled photonic acquisition experiment, it shows that prototype-compressed retrieval can be applied to real industrial surface-defect images after excluding pixel-level mask files. The result is nuanced: ConvNeXt-Tiny kNN gives the highest overall Top-1, whereas MP-GDM improves Macro Top-1 under class imbalance. This is exactly why the experiment is reported as a sanity check rather than as a new SOTA benchmark.

The novelty of MP-GDM lies in how it is used in the retrieval system rather than in inventing clustering itself. Standard single-centroid or few-shot prototypes summarize a class for classification [37], and clustering/approximate search has long been used for efficient retrieval [38,39]. MP-GDM instead acts as a validation-selected reference-gallery compression layer after metric training. It retains multiple normalized class prototypes so that multi-modal material appearances are not collapsed into a single centroid, and it evaluates prototype count and aggregation through Top-k, memory, and latency. The feature-space visualization and large-gallery scaling analysis support this role: MP-GDM reduces the operational cost of global retrieval while preserving enough class-mode structure for ranked material-surface search.

Similarly, RAEF should be viewed as a reliability-aware candidate family, not as a fully probabilistic uncertainty model. Margin confidence, entropy confidence, and branch agreement are practical uncertainty proxies inspired by calibration and uncertainty literature [40,41,42,43]. The calibration curves show that these cues are informative but imperfect, and the disagreement analysis shows that branch disagreement can indicate ambiguity without guaranteeing improvement. This conservative interpretation is important for optical inspection: a reliability module should help users understand when evidence sources conflict, but it should not be presented as a universal adaptive rule unless it is validated across more acquisition conditions and datasets.

The current study also has clear limitations. The public RGB texture/material benchmarks and the Magnetic Tile image-folder check are reproducible proxies for surface image analysis, not controlled photonic experiments. They do not systematically vary illumination spectrum, polarization, sensor response, exposure, viewing geometry, microscopic scale, or computational-imaging reconstruction pipeline. Future work should test RAGL-MR under controlled optical acquisition, microscopic inspection, industrial line-scan or area-scan systems, and computational imaging outputs. The failure cases suggest several technical directions: class-aware calibration for fiber-like categories, local-patch quality scoring for repeated background structures, illumination or color normalization for reflective classes, stronger foundation backbones where image-level transfer is decisive, and learned lightweight branch gating that is validated across multiple splits rather than tuned to one diagnostic setting.

Finally, deployment efficiency remains an engineering concern. Prototype compression makes global retrieval compact, but multi-scale local patch evaluation is still expensive because patch extraction, embedding, nearest-neighbor scoring, and score-cache writing are partly sequential. Parallel query scheduling, patch prefetching, and batched cache writes could reduce this bottleneck without changing the algorithmic claim. Thus, the present paper should be read as an evidence-aware retrieval framework and validation study; a production optical inspection system would still require controlled acquisition validation and systems-level optimization.

6. Conclusions

This paper presented RAGL-MR, a reliability-aware global-local metric retrieval framework for optical image-based texture and material surface recognition. The method combines multi-scale local patch retrieval, MP-GDM global prototypes, and validation-selected reliability-aware fusion candidates. Across DTD, MINC-2500, FMD, and KTH-TIPS2-b, the results show that the framework is most valuable as an evidence-aware ranked retrieval system. Rather than claiming universal dominance over closed-set classifiers or frozen foundation features, RAGL-MR provides complementary advantages for optical material surface analysis: inspectable patch-level surface evidence, prototype-compressed global retrieval, reference-gallery extensibility, low-confidence recovery analysis, branch-disagreement diagnostics, and validation-controlled fusion. The extended ConvNeXt, CLIP, and DINOv2 controls further clarify the scope of the method: strong classifiers and foundation features remain competitive or superior on some closed-set or image-level retrieval metrics, while RAGL-MR focuses on retrieval evidence and reliability-oriented analysis.

Future work will focus on three directions. First, controlled optical imaging experiments should examine illumination, wavelength, polarization, viewing geometry, and sensor-domain variation rather than relying only on public texture/material image benchmarks. Second, broader matched comparisons with stronger closed-set classifiers, foundation retrieval models, and domain-adaptation strategies will clarify the relationship between image-level accuracy and evidence-aware retrieval. Third, lightweight deployment should combine more efficient local-patch evaluation, parallel query scheduling, and learned reliability calibration so that the framework can better support practical photonic visual sensing and material inspection.