Hyperspectral Imaging Combined with Deep Learning for the Detection of Mold Diseases on Paper Cultural Relics

Abstract

Mold contamination is one of the critical factors threatening the safety of paper-based cultural relics. Current detection methods rely predominantly on offline analysis, facing challenges such as low efficiency and limited real-time accuracy, which hinder their effectiveness in meeting the technical requirements of cultural heritage preventive conservation. This study proposes a hyperspectral imaging (HSI)-deep learning integrated fungal segmentation framework for deterioration detection in paper-based artifacts. Firstly, the HSI data was reduced to three dimensions via Locally Linear Embedding (LLE) manifold learning to construct 3D pseudo-color imagery, effectively preserving discriminative spectral features between fungal colonies and substrates while eliminating spectral redundancy. Secondly, a hybrid architecture synergizing Feature Pyramid Networks (FPN) with Vision Transformers was developed for semantic segmentation, leveraging CNN’s local feature extraction and Transformer’s global context modeling to enhance fungal signature saliency and suppress background interference. Innovatively, a dynamic sparse attention mechanism is introduced, optimizing attention allocation through the TOP-K algorithm to screen regions richer in mold information spatially and spectrally, thereby improving segmentation accuracy. Semantic segmentation experiments were conducted on papers infected with different molds. The results demonstrate that the proposed method achieves excellent performance in mold segmentation, providing technical support for mold detection and preventive conservation of cultural relics.

1. Introduction

Due to the limitations of current monitoring and preservation technologies, the deterioration of cultural relics caused by biological factors (such as pests and fungi) has become increasingly prominent, posing a major threat to the long-term preservation, display, and utilization of cultural relics. This issue is particularly severe for paper-based artifacts. Fungi, characterized by their widespread distribution, rapid growth, resilience, and ease of airborne transmission [1], can proliferate rapidly and erode the surface of artifacts under suitable conditions. The secretion of metabolites and enzymes by molds directly degrades the fibrous structure of paper artifacts, significantly diminishing their historical and artistic worth.

Currently, traditional fungal detection methods primarily rely on visual inspection and microscopic examination. However, these methods are inadequate for early detection of fungal infection and accurate assessment of fungal distribution and infection extent. Although offline detection techniques such as fluorescence spectroscopy [2] and scanning electron microscopy [3] can effectively detect fungi, the equipment required for these techniques is complex, precluding online automated detection. Since the 1990s, imaging spectroscopy technology has been introduced into the field of cultural relic preservation. Although traditional spectral analysis [4,5] can be used for the detection of microbial pests such as fungi, it has significant shortcomings in terms of precise identification, detection of minute areas, and quantitative analysis.

With the evolution of spectral technology from multispectral to hyperspectral imaging, the quality of spectral information obtained has significantly improved. Hyperspectral technology has been widely applied in food safety inspection. For example, Farrugia J et al. [6] used hyperspectral imaging combined with PCA to detect fungi in milk, while Long Y et al. [7] employed Raman hyperspectral imaging technology for the detection and identification of fungi in corn. In the field of cultural relic inspection, hyperspectral imaging technology has also demonstrated tremendous potential. Lan R et al. utilized hyperspectral imaging technology to detect contaminants on the surface of cultural relics and analyze their materials, providing in situ analysis and assessment methods for the aging degree of paper and textile fibers in painting and calligraphy artifacts [8]. Wang S et al. successfully achieved texture, image, and color restoration of artifacts by acquiring and processing hyperspectral images of paintings contaminated with fungi [9].

Analyzing the distribution and species of mold from hyperspectral data of cultural relics is similar to object recognition in other fields. Currently, object recognition primarily employs deep learning algorithms based on CNNs, including target detection methods represented by the YOLO series [10] and semantic segmentation algorithms represented by the U-Net series [11]. Semantic segmentation, being pixel-level detection, delivers finer recognition precision. To enhance the accuracy of semantic segmentation, many methods adopt skip connections [12] to integrate low-level and high-level features. Simultaneously, attention mechanisms [13] are introduced during the network’s feature extraction stage to focus on information-rich regions and improve segmentation performance. However, the surface of mold on paper artifacts often contains indistinguishable impurities, ink marks, and other elements that share similar characteristics with the mold of interest. Due to the inherent local nature of CNN-based methods, they have limitations in capturing global information from images, which can easily lead to misidentification and omission of mold. The Transformer model has demonstrated exceptional performance in natural language processing tasks, and its variant, the Vision Transformer (ViT) [14,15], has achieved state-of-the-art results in image classification tasks, fully demonstrating the significant potential of Transformer in computer vision tasks. Consequently, numerous ViT-based methods have been applied to the semantic segmentation of objects. The SETR model, which employs ViT as its encoder, validated the capability of a pure Transformer in semantic segmentation, though its segmentation accuracy still requires improvement. This occurs because the self-attention mechanism in Transformer inherently functions as a low-pass filter, which tends to suppress high-frequency information. When targets are indistinct, this may lead to the loss of mold-related information.

In the field of semantic segmentation, several studies [16,17] have attempted to integrate the strengths of CNNs and Transformers into a unified model by incorporating the local properties of convolution into Transformers. This integration aims to simultaneously capture local contextual information and build global dependencies. Some approaches introduce the local characteristics of convolution [18] into Transformers to enhance their ability to capture local information, thereby retaining more detailed features. However, under conditions of complex background interference and subtle mold characteristics, these methods still face significant challenges.

Based on the above analysis and research, this manuscript designs a mold segmentation network for paper artifacts that combines a CNN and Transformer. The overall structure of the network consists of two major parts: an encoder and a decoder. Inspired by HiFormer [19], this paper employs a Feature Pyramid Network (FPN) [20,21] to generate multi-level features at different resolutions, which are then unidirectionally fed into a multi-stage Transformer for fusion. Multi-level features of different resolutions contain different emphases of mold information, and they serve as inputs to various stages of the Transformer in the encoder, enabling the entire encoder to extract more effective feature information. Furthermore, this paper introduces the TOP-K algorithm into the self-attention mechanism and proposes a dynamic sparse attention module. This module performs variance sorting on the channels and spatial dimensions of the feature maps to select channels and regions with richer mold information, while reducing the interference of background noise, which is beneficial for mold segmentation.

In summary, the main contributions of this paper are as follows:

(1) A mold segmentation network that integrates multi-level features is proposed. Multi-level CNN features generated by the FPN are fused in the Transformer layers, combining the ability of CNNs to capture detailed features with the ability of transformers to suppress background noise, thereby enhancing mold feature information.

(2) A dynamic selection sparse self-attention module is proposed, which selects important mold features from both spatial and channel dimensions and suppresses redundant background noise.

2. Related Work

2.1. Dimension Reduction

In the field of data dimensionality reduction, traditional methods such as Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) struggle to reveal the intrinsic manifold structure of hyperspectral data, thereby limiting their practical application performance. In recent years, manifold learning algorithms have emerged as effective approaches for discovering low-dimensional manifold structures within high-dimensional data. Among these, Locally Linear Embedding (LLE) [22] operates under the assumption that data exhibits linear characteristics locally and preserves this local structure during dimensionality reduction; Laplacian Eigenmaps [23] maintain local neighborhood relationships by constructing adjacency graphs; while Isometric Mapping (ISOMAP) [24] preserves the global manifold structure using geodesic distances.

In comparison, methods like t-SNE [25], despite offering excellent visualization, often fail to maintain global structures, making them unsuitable as feature inputs for downstream tasks. This study ultimately selects the Locally Linear Embedding algorithm due to its theoretical simplicity, strong reproducibility, and the close alignment of its core concept with our focus on the “local similarity” hypothesis. Moreover, as a classical algorithm, it provides a solid theoretical foundation for our research.

2.2. Vision Transformer

The Transformer was originally designed for natural language processing tasks, and its outstanding performance has demonstrated its tremendous potential. Unlike the local convolution operations of Convolutional Neural Networks (CNNs), the built-in self-attention mechanism of the Transformer can effectively capture global dependencies, as shown in Figure 1.

In recent years, Transformers have been widely applied to vision tasks. Dosovitskiy et al.[26] proposed Vision Transformer (ViT), which, for the first time, treats an image as a sequence of position vectors fed into a Transformer for image classification tasks. Given the excellent performance of ViT, Ramachandran Prajit et al. [27] introduced a Stand-Alone self-attention network that replaces convolutions with an attention mechanism. However, due to the lack of translation equivariance and locality in the Transformer architecture, it performs poorly on smaller datasets. To enhance the generalization capability of Transformers, Ze Liu et al. [28] proposed a novel local attention mechanism that establishes global relationships by shifting windows along spatial dimensions and dynamically attends to neighboring elements, thereby improving the ability to extract local information. Nevertheless, the self-attention mechanism inherently exhibits low-pass filtering characteristics, making it difficult to recognize small or inconspicuous molds.

2.3. Segmentation Network Based on the Combination of ViT and CNN

CNNs excel at extracting local features, but they are prone to introducing noise when extracting feature information in convolutional layers. On the other hand, the inherent self-attention mechanism of Transformers can model global relationships and effectively suppress background noise interference. Therefore, some approaches attempt to combine CNNs with Transformers to equip the network with the ability to model long-range dependencies and focus on local information simultaneously.

Wang, WK et al. [29] proposed a multigranularity hybrid CNN-ViT model based on external tokens and cross-attention for PolSAR image classification. A CNN-based external feature extractor was designed to extract local features from the PolSAR image and ViT can focus on global features. A segmentation method [30] that fuses CNN and ViT is proposed to address the problems of large differences in shape and size of tumor regions of breast ultrasound images that lead to difficulty in segmentation, limitations in long range dependency and spatial correlation in convolutional neural network (CNN) modeling, and the huge amount of data required by vision Transformer (ViT). Global and local detail features were extracted using a modified Swin Trans former module and a CNN encoder module based on deformable convolution, respectively.

Although these methods inherit the local characteristics of CNNs while maintaining the global advantages of Transformers, they still face challenges from inconspicuous features and complex backgrounds. Unlike the aforementioned methods, this paper employs a CNN feature pyramid to generate feature maps at different scales, preserving more local information through a top-down and bottom-up pyramid structure. Then, feature maps of different resolutions are used as inputs to different Transformer layers, achieving the fusion of CNN features within the Transformer. Furthermore, this paper adopts a dynamic sparse attention module to select regions richer in mold information both spatially and across channels, while suppressing background interference information.

3. Methodology

3.1. Dimensionality Reduction with Locally Linear Embedding (LLE)

The basic assumption of the LLE algorithm is that data is linear within small local regions, meaning that a data point can be linearly reconstructed by its nearest neighbors. For example, given a sample ${\mathit{x}}_1$, we use the K-nearest neighbors approach to find the k closest samples ${\mathit{x}}_2$, $\cdots$, ${\mathit{x}}_{\mathit{k}+1}$, in its original high-dimensional neighborhood. It is assumed that ${\mathit{x}}_1$ can be linearly represented by ${\mathit{x}}_2$, $\cdots$, ${\mathit{x}}_{k+1}$ as shown in Equation (1):

$${\mathit{x}}_1=q_{12}\times {\mathit{x}}_2+q_{13}\times {\mathit{x}}_3+\cdots +q_{1k+1}\times {\mathit{x}}_{k+1}$$

(1)

where $q_{12}$, $q_{13}$, $\cdots$, $q_{1k+1}$ are the weight coefficients. After dimensionality reduction using LLE, it is hoped that the projection of $x_1$ in the low-dimensional space, denoted as $y_1$, will maintain the same linear relationship with the projections of $x_2$, $x_3$, $\cdots$, $x_{k+1}$ denoted as $y_2$, $y_3$, $\cdots$, $y_{k+1}$ respectively, as shown in Equation (2):

$${\mathit{y}}_1=q_{12}\times {\mathit{y}}_2+q_{13}\times {\mathit{y}}_3+\cdots +q_{1k+1}\times {\mathit{y}}_{k+1}$$

(2)

The original high-dimensional sample has a corresponding low-dimensional projection, where, meaning that the LLE dimensionality reduction algorithm reduces the vector dimension while preserving local relationships unchanged. The main parameters of the LLE algorithm include the target dimension and the neighborhood size. The specific principles and computational process of the algorithm can be found in reference [22]. In this paper, we use this algorithm to reduce the high-dimensional spectrum to a three-dimensional space using LLE, resulting in a vector as shown in Equation (3):

$$\mathit{U}=\mathrm{LLE}(\mathit{W},d,k)=\{{\overline{w}}_1,{\overline{w}}_2,\cdots {\overline{w}}_c\}\in {\mathit{R}}^{C\times 3}$$

(3)

$\mathrm{LLE}(\mathit{W},d,k)$ represents the LLE dimensionality reduction operation, $d$ denotes the target dimension, which is 3 in this case, and $k$ denotes the neighborhood size. $U\in R^3$ represents the result of reducing the high-dimensional vector to three dimensions.

The following figure shows the spectral curves of Aspergillus niger, Penicillium citrinum, ink, and paper background extracted from the spectral data of cultural relics. Each spectral curve contains 300-dimensional spectral features. As shown in Figure 2a, there are significant differences in spectral features among different targets or backgrounds. High-dimensional features are complex and contain a lot of redundancy. After LLE dimensionality reduction, the resulting three-dimensional spectral features are shown in Figure 2b. In the three-dimensional space, the differences in spectral features among different targets or backgrounds are effectively highlighted. Additionally, these three-dimensional spectral features can be used as the red, green, and blue color channels to construct false-color spectral images of paper cultural relics.

3.2. Mold Semantic Segmentation

3.2.1. Overall Network Architecture

Inspired by the respective outstanding performances of CNNs and Transformers in semantic segmentation tasks, we have designed a mold detection semantic segmentation network that unifies the ideal attributes of both into a single architecture. The overall architecture is shown in Figure 3.

The network adopts an encoder-decoder approach. The encoder is mainly divided into two parts: the CNN module and the Transformer module. ResNet50 and MaxViT [31] are used as the backbone networks for the CNN and Transformer, respectively. ResNet50 is a milestone architecture widely adopted in computer vision. Its residual connections effectively alleviate the gradient vanishing problem in deep networks, making it a benchmark model for numerous tasks. Leveraging pre-trained weights from large-scale datasets like ImageNet, ResNet50 can extract hierarchical features with strong generalization capabilities, laying a solid foundation for downstream tasks. As an innovative hybrid architecture, MaxViT maintains local feature extraction through convolutional operations while incorporating Transformer modules to establish global dependencies, effectively overcoming the limitations of traditional CNNs in long-range modeling. During the encoding phase, feature maps of different scales and dimensions are first generated through four levels of convolutional layers, each focusing on different local information, thereby forming hierarchical representations. Subsequently, these hierarchical representations are fed into different Transformer layers within the encoder to complete the network’s encoding operation and achieve the fusion of multi-level features.

In the decoding phase, the network consists of four levels of Transformer layers. Each Transformer layer is composed of a different number of MaxViT modules stacked together, with the specific configuration parameters shown in

Table 1. MaxViT Parameters for Each Transformer Layer.

	Layer	Configuration Parameters	Output Size
Input	-	-	3 × W × H
encoder	Layer	configuration parameters	output size
	S1	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Grid}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$64\times \frac{W}{4}\times \frac{H}{4}$
	S2	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Grid}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$128\times \frac{W}{8}\times \frac{H}{8}$
	S3	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Dynamic}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$256\times \frac{W}{16}\times \frac{H}{16}$
	S4	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Dynamic}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$512\times \frac{W}{32}\times \frac{H}{32}$
decoder	Layer	configuration parameters	output size
	Z1	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Grid}-MSA(G=8,H=2)\end{array}\right\}\times 1$	$256\times \frac{W}{16}\times \frac{H}{16}$
	Z2	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Grid}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$128\times \frac{W}{8}\times \frac{H}{8}$
	Z3	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Grid}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$64\times \frac{W}{4}\times \frac{H}{4}$
	Z4	$\left\{\begin{array}{c}MBConv(E=4,R=4)\\ \mathit{Window}-MSA(W=8,H=2)\\ \mathit{Grid}-MSA(G=8,H=2)\end{array}\right\}\times 2$	$64\times \frac{W}{4}\times \frac{H}{4}$

.

The last Transformer layer of the encoder contains rich contextual information, serving as a bridge between the encoder and the decoder. Meanwhile, during the downsampling process in the encoder, some spatial information of the mold may be lost. To address these issues, the outputs of the four encoding layers are skip-connected to the four decoding layers in the network structure, forming a complete mold detection network.

3.2.2. Feature Pyramid Module

The pyramid structure has demonstrated excellent performance in improving model capabilities and is therefore widely used in tasks such as object detection and semantic segmentation. Figure 4a employs the last layer for prediction. However, due to the convolutional downsampling operation, information about small mold regions may be lost, resulting in poor segmentation performance for inconspicuous mold. Figure 4b utilizes high-level feature maps for multi-scale prediction. A single high-level feature lacks both sufficient semantic information and adequate utilization of spatial information from low-level feature maps, which is crucial for detecting small mold regions.

Based on ResNet50, this paper designs a Feature Pyramid Network (FPN) structure, as shown in Figure 4c. Unlike traditional pyramids, FPN fuses multi-scale information in a top-down and bottom-up manner, allowing both large and small objects to have appropriate feature representations at corresponding scales. The original image A undergoes top-down convolution to generate multi-layer features. The bottom-layer features are upsampled and then fused with other features to generate hierarchical CNN features.

In the Transformer layers of the encoder, the convolutional features R1 are used as input instead of the original image, while R2, R3, and R4 are respectively fed into different levels of the Transformer network, providing detailed features of the mold for the Transformer layers. The mathematical process is shown in Equation (4):

$$R_{i+1}=\mathit{Conv}\left(\mathit{Concat}\right(\mathit{UpSampling}\left(L_i\right),L_{i-1}\left)\right),i=0,1,2,3$$

(4)

“UpSampling(.)” represents upsampling, “Conv(.)” represents convolution, and “Concat(.)” represents concatenation.

3.2.3. MAXViT Module

The MaxViT module is a core component of the Transformer network, with its structure illustrated in Figure 5. It is mainly divided into two parts: the MBConv module and the Multi-Axis Self-Attention (MaxSA) module. Research has shown that combining local and global approaches helps improve the model’s generalization ability. Therefore, before applying the MaxSA module, we preprocess the image information using the MBConv module.

The MBConv module introduces local characteristics through convolution operations and utilizes the Squeeze and Excitation (SE) network to assign higher weights to mold feature information. Additionally, the MBConv module employs DepthWise Conv (DWC) instead of traditional positional encoding, eliminating the need for explicit positional encoding in the network, reducing the number of network parameters, and improving the network’s training speed.

C is the feature tensor, which serves as the input to the MAXViT module. The mathematical process is shown in Equation (5).

In the second stage, the MaxSA module is applied to extract global information. The MaxSA module decomposes the traditional self-attention mechanism into two sparse forms: Block Self-Attention (BSA) and Grid Self-Attention, connected in a serial manner with the BSA module followed by the Grid Self-Attention module. This reduces the quadratic complexity of self-attention computation to linear without sacrificing globality. C1 is the output of the MBConv module, BSA(.) represents block self-attention, FFN(.) represents the feedforward neural network, C2 is the output of the features after passing through the block self-attention module, GSA(.) represents grid self-attention, and C3 is the output of the MAXViT module. The mathematical processes are shown in Equations (5)–(7):

$$C^1=C+\mathit{PROJ}\left(\mathit{SE}\right(\mathit{DWC}\left(\mathit{Conv}\right(\mathit{BN}\left(C\right)\left)\right)\left)\right)$$

(5)

$$C^2=C^1+\mathit{BSA}\left(C^1\right)+\mathit{FFN}(C^1+\mathit{BSA}(C^1\left)\right)$$

(6)

$$C^3=C^2+\mathit{GSA}\left(C^2\right)+\mathit{FFN}(C^2+\mathit{GSA}(C^2\left)\right)$$

(7)

3.2.4. Dynamic Sparse Attention Module

Traditional self-attention calculates the relevance of all position vectors and extracts important information based on the computed attention scores, capturing global dependencies between position vectors. However, this mechanism entails an excessively large computational load. The sparse attention mechanism serves as a key optimization strategy to address the prohibitively high computational complexity and substantial memory consumption of the standard Transformer model’s self-attention. Sparse attention computes correlations only between a pre-selected, limited number of positional pairs. This significantly reduces the number of attention pairs that need to be calculated, enabling the model to handle longer sequences previously beyond its reach.

The sparse attention model proposed by Jiang M et al. [32] sparsifies the feature block Query and feature block Key defined in the self-attention module through a credit matrix generated by the pre-output layer, followed by similarity modeling between the two sparsified feature blocks, thereby significantly reducing computational resource consumption. The method for selecting sparse positional pairs is crucial for sparse attention mechanisms. Sun Z et al. [33] defined an Adaptive-Reciprocal Nearest Neighbors (A-RNN) relationship, which achieves flexibility through adaptive threshold learning during neighbor identification while ensuring reliability of neighbor relations via a reciprocity mechanism. Sun B et al. [34] integrated dynamic sparse attention through a bi-level routing mechanism, achieving context-aware computational resource allocation with enhanced adaptability. This approach not only optimizes semantic feature extraction but also ensures feature quality.

In this study, most position vectors primarily contain background information rather than mold information, leading to attention maps generated by self-attention focusing more on background regions, with small and weak mold features being overwhelmed by redundant background information. To address these issues, we introduce the TOP-K algorithm into the multi-axis attention mechanism and design an effective Dynamic Sparse Self-Attention module. The dynamic sparse attention mechanism selects K position vectors with higher scores in both spatial and channel dimensions to compute attention, filtering out some redundant background information and highlighting important mold features, while also reducing the computational load of self-attention. The module details are shown in Figure 6, where low-scoring gray areas are filtered out, and high-scoring bright areas are retained.

Unlike the traditional attention mechanism, the number of position vectors involved in dynamic sparse attention computation is limited by the window size of the MaxSA mechanism. In our experiments, the window size is set to 8 × 8. The output of the MBConv module $C^1\in {\mathbb{R}}^{B\times C\times W\times H}$, C1, is flattened to $X\in {\mathbb{R}}^{B\times \frac{W}{8}\times \frac{H}{8},8\times 8,C}$ serve as the input to the sparse attention module. It is then linearly mapped to the query Q, key K, and value V through three matrices: $W_q\in {\mathbb{R}}^{(C\times C)}$, $W_k\in {\mathbb{R}}^{(C\times C)}$, $W_v\in {\mathbb{R}}^{(C\times C)}$as shown in the mathematical process in Equation (8):

$$Q=X{W}_q,K=X{W}_k,V=X{W}_v$$

(8)

Based on the Q, K, and V obtained from the linear mapping, combined with the Top-K algorithm, we retain position vectors and channels rich in mold information while suppressing the interference of background information. Given the nature of discrete data, variance can reflect the degree of dispersion of the data; the larger the variance, the greater the data fluctuation, and the richer the mold information it contains. Therefore, we calculate the variance along the channel dimension, select K position vectors with the largest variance values based on their ranking, and simultaneously calculate the variance along the spatial dimension to select K most important channels. In the dynamic sparse attention module, we first calculate the channel variance and spatial variance on the query Q to obtain the coordinate indices of K position vectors and channels. Using these coordinate indices, we select the most important position vectors and channels from Q and K to generate Q1 and K1, and select important position vectors from V to generate V1. To ensure that Q, K, and V correspond to the same position vectors and channels at the same positions, we calculate the variance values only once in Q to determine the position vector indices and channel indices. We then use the obtained Q1, K1, and V1 to compute the attention. SoftMax(.) represents the SoftMax function, and C represents the channel dimension of Q1 and K1. The formula is shown in Equation (9):

$$\mathit{Attention}(Q^1,K^1,V^1)=\mathit{SoftMax}\left(\frac{Q^1{K}^{1T}}{\sqrt{C}}\right)V^1$$

(9)

3.3. Experimental Design

3.3.1. Experimental Procedure

The overall experimental process is shown in Figure 7. A hyperspectral camera is used to scan paper containing mold to obtain three-dimensional spectral data. The LLE (Locally Linear Embedding) linear manifold learning algorithm is employed to reduce the high-dimensional spectral data to 3 dimensions. These three-dimensional features are then used as the red, green, and blue channels of the image to synthesize a false-color image. Subsequently, the semantic segmentation algorithm proposed in this paper is applied to segment the mold in the false-color image.

3.3.2. Spectral Data Acquisition

In this experiment, the portable hyperspectral imaging system iSpecHyper-VS1000 from Lesen Optics was used to collect hyperspectral data of mold spots on paper artifacts in reflection mode. The acquisition setup is shown in Figure 8. The setup consists of a computer (equipped with acquisition software developed by Lesen Optics), a portable hyperspectral imaging system, two 1000 W halogen light sources, and a Canon RF24 lens (Canon Inc., Tokyo, Japan). The spectral range is 400–1000 nm, with a resolution of 2.8 nm. Parameters are listed in

Table 2. Parameters of the iSpecHyper-VS1000 Hyperspectral Imaging System.

Main Technical Indicators	Indicator Parameters	Main Technical Indicators	Indicator Parameters
Spectral Range	400 nm~1000 nm	AD Dynamic Range	12 bits
Spectral Resolution	<3 nm	Image Resolution	1920 × 1200
Number of Spectral Channels	300	Lens	Canon 24 mm
Detector	CMOS/InGaAs (TE Cooled)	Pixel Size	5.86 μm × 5.86 μm

.

Since deep learning-based methods require a substantial number of samples while cultural artifacts are extremely precious, the mold-infected paper samples used in this study were artificially cultivated by staff at the Chongqing Three Gorges Museum based on mold types commonly found infesting paper artifacts in their collection. The study selected Xuan paper with material properties similar to those of cultural artifacts, and employed common mold species including Aspergillus niger, Penicillium citrinum, and Cladosporium cladosporioides. The specific procedure involved using a disposable sterile swab to perform unidirectional gentle swabbing from pre-cultured molds, transferring the contaminated swab to a glucose culture medium for incubation. After three days of cultivation, the culture broth was diluted with sterile deionized water and subsequently inoculated onto the sample paper. For each type of mold, 30 samples were prepared. Following hyperspectral scanning, the samples were cropped, with each original sample divided into 12 sub-samples, resulting in 360 sub-samples for each of the three molds. Each sub-sample image has a resolution of 256 × 256 pixels and was split into training, validation, and test sets in a ratio of 7:2:1. The Ground Truth of mold distribution areas was annotated by museum staff through visual inspection combined with microscopic observation. Identification of the three mold types was performed and compared with other models in a comparative experimental analysis. Finally, an ablation study was conducted on one of the datasets for the dynamic sparse attention module to verify its effectiveness.

Each pixel in a hyperspectral image contains a complete set of spectral data, which includes the absorption or reflection information at 300 different and continuous wavelengths at that pixel location, as shown in Figure 9. In this hyperspectral image, the spectral data of pixels belonging to the same type exhibit great similarity, while the spectral data of pixels belonging to different types show significant differences. Therefore, using spectral data provides a scientific basis for classifying hyperspectral images into different categories.

3.3.3. Experimental Configuration

The network model proposed in this paper is implemented based on the PyTorch 2.3.1 framework, and all training processes are completed on an AutoDL server equipped with NVIDIA 3090 series GPUs. During training, the initial learning rate of the model is set to 0.01, and the model is trained using an SGD optimizer with a decay rate of 0.001 and a momentum of 0.9. Additionally, the learning rate is decayed according to Equation (10) during the training process.

$$\mathit{lr}\_=\mathit{base}\_\mathit{lr}\times ({1.0-\frac{\mathit{iter}\_\mathit{num}}{\mathit{max}\_\mathit{iter}})}^{0.9}$$

(10)

The base learning rate (base_lr) is set to 0.01, iter_num represents the current training iteration, and max_iter is the maximum number of training iterations. To address the issues of class imbalance among mold categories and foreground-background imbalance in the dataset, the model’s loss function adopts a combined form, as shown in Equation (11):

$$\mathit{Loss}=0.4\times \mathit{Loss}\_\mathit{ce}+0.6\times \mathit{Loss}\_\mathit{dice}$$

(11)

where Loss_ce is the cross-entropy loss function, and Loss_dice is the Dice similarity coefficient loss function.

For each dataset, the images in both the training and test sets are resized to a uniform resolution of 512 × 512. The model is trained for 150 epochs on each of the three datasets, with a batch size of 24. The mean Intersection over Union (mIoU) metric is used to evaluate the model’s segmentation performance.

4. Experiments and Result Analysis

4.1. Experimental Comparative Analysis

To verify the superiority of the proposed model, comparative analyses were conducted against state-of-the-art segmentation networks including HiFormer [19], MaxVIT [31], ResNet50, DeepLabV3 [35], and TransUnet [36]. As quantitatively demonstrated in

Table 3. Comparison data of different models on three defect data.

Models	Parameter Count (M)	FLOPs	Aspergillus niger	Penicillium citrinum	Cladosporium cladosporioides
			mIoU (%)	mIoU (%)	mIoU (%)
ResNet50	35.3	36.3 G	87.7	83.3	85.4
Deeplabv3	39.6	40 G	92.4	82.5	86.3
MaxVIT	26.8	9.2 G	94.3	80.6	88.4
TransUnet	93.2	31.5 G	96.3	89.6	87.8
HiFormer	34.2	17.4 G	96.1	90.3	87.9
Ours	42.8	15.7 G	96.8	90.1	89.3

  (where bold values indicate the best results for each metric), the proposed architecture achieves significant improvements in fungal segmentation accuracy. The hybrid architectures (TransUnet, HiFormer, and Ours) consistently outperform pure CNN or Transformer models across all three fungal species, demonstrating the efficacy of integrating multi-scale CNN features with global Transformer attention. For Penicillium citrinum, HiFormer achieves the highest mIoU (90.3), closely followed by Ours (90.1), significantly surpassing pure CNN models like ResNet50 (83.3) and pure Transformer MaxVIT (80.6). In Aspergillus niger, Ours sets a new benchmark (96.8 mIoU), outperforming even TransUnet (96.3) and MaxVIT (94.3). For Cladosporium cladosporioides, Ours again leads (89.3), with MaxVIT (88.4) and HiFormer (87.9) trailing slightly.

The visual results for the semantic segmentation of different types of mold are shown in Figure 10. Quantitative experimental data and semantic segmentation results demonstrate that all semantic segmentation networks can accurately identify mold and background regions in the Aspergillus niger dataset, as shown in rows 1 to 4 of Figure 10. Overall, our method achieves the best performance, with an mIoU at least 0.5% higher than other methods. In the Penicillium citrinum dataset, many molds share similar spectral characteristics with the paper background. CNN-based models like ResNet50 and DeepLabV3 tend to misidentify background regions resembling molds as actual molds and fail to detect low-contrast molds. In contrast, models incorporating global features—MaxVIT, HiFormer, TransUnet, and our proposed network—effectively suppress background interference and accurately identify molds by establishing long-range dependencies. HiFormer uses only two feature maps with the richest semantic and positional information during decoding, while our model employs multi-scale feature fusion. Although this preserves more mold details, it introduces partial noise. HiFormer surpasses our model by 0.2% in segmentation accuracy, with results illustrated in rows 5 to 8 of Figure 10. Cladosporium molds are sparsely distributed with varying infection depths, making detection challenging. Our model retains multi-scale information and demonstrates exceptional detection capabilities on the Cladosporium dataset, achieving a 1.5% higher mIoU than TransUnet and outperforming HiFormer by 1.4%. Segmentation results are shown in rows 9 to 10 of Figure 10.

4.2. Ablation Experiment

To further demonstrate the effectiveness of the proposed dynamic sparse attention module in our network, we conducted an ablation experiment on the Cladosporium dataset focusing on this module.

Firstly, we divided the encoding layers of the network into four levels: 1, 2, 3, and 4 in sequence. We then replaced the standard self-attention module with the dynamic sparse attention module in a reverse order, layer by layer. The experimental results are shown in

Table 4. Comparison of dynamic sparse attention modules using different levels on Penicillium citrinum.

Model	Stage				Param	Flops	Penicillium citrinum
	1	2	3	4			mIoU
Ours					43 M	15.8 G	86.9
				√	43 M	15.7 G	89.5
			√	√	43 M	15.7 G	90.1
		√	√	√	42 M	15.6 G	90.0
	√	√	√	√	42 M	15.6 G	88.7
MaxUnet					26 M	9.2 G	82.4
				√	26 M	9.1 G	83.9
			√	√	26 M	9.0 G	87.3
		√	√	√	25 M	8.9 G	86.3
	√	√	√	√	25 M	8.9 G	84.2

Note: ‘√’ indicates that the self-attention module is replaced by the dynamic sparse attention module.

. As indicated in Table 4, the network incorporating the dynamic sparse attention mechanism exhibits better segmentation performance compared to the network based on the self-attention mechanism. The introduction of the dynamic sparse attention module at the fourth, third, second, and first levels of our network improves the performance by 2.6%, 3.2%, 3.1%, and 1.8%, respectively. This demonstrates the effectiveness of the dynamic sparse attention module. An analysis of the results reveals that the introduction of the dynamic sparse attention module at the third and fourth levels yields the best results. This is because the feature maps generated in the early stages of the network contain rich detailed information about the mold, and the introduction of the dynamic module may result in the loss of some important details, reducing the accuracy of weak mold recognition. In contrast, the feature maps generated in the later stages of the network contain semantic information about the mold, which is easily interfered with by background noise.

Meanwhile, more significant position vectors are selected both in the channel and spatial dimensions to compute attention, reducing the model’s complexity. Finally, we investigate the impact of the hyperparameter K on the network’s performance. We set K to 24, 32, 40, 48, 56, and 64, respectively. As shown in

Table 5. Comparison of different K values of dynamic sparse attention modules on Penicillium citrinum.

Model	Parameters	Complexity	K	Penicillium citrinum
				mIoU
Ours	41 M	15.5 G	24	87.6
	42 M	15.6 G	32	88.9
	42 M	15.6 G	40	89.6
	43 M	15.7 G	48	89.2
	43 M	15.7 G	56	90.1
	43 M	15.8 G	64	87.1
MaxUnet	24 M	8.8 G	24	85.1
	24 M	8.9 G	32	84.8
	25 M	8.9 G	40	83.9
	25 M	9.0 G	48	84.7
	26 M	9.0 G	56	85.8
	26 M	9.1 G	64	85.2

, our model achieves the best performance when K is 56. If K is too large, weak mold information with low contrast is easily interfered with by background information. Conversely, if K is too small, position vectors containing useful information are lost, reducing the model’s ability to recognize mold.

4.3. Application Testing

To validate the effectiveness of the proposed method, a set of paper artifacts infected with Cladosporium and their corresponding spectral data, provided by the Chongqing Three Gorges Museum, were collected and subjected to mold segmentation testing using our approach. Due to the inability to conduct prolonged testing on cultural artifacts, fine-grained annotations were not available, making it difficult to compute quantitative metrics. However, based on the visual segmentation results, the primary areas of mold distribution were largely accurately identified, as shown in Figure 11b. That said, a small number of mold instances located within ink stains were missed, likely because the semantic segmentation process, which relies on contextual information, was interfered with by local ink patterns. Examples of such cases are illustrated in Figure 11f,g. Overall, the method demonstrated satisfactory performance in identifying mold categories, their distribution on paper artifacts, and the extent of infection, suggesting its potential as a technical tool for monitoring museum collections.

5. Conclusions

In response to the early detection needs of mold infestation on paper artifacts, this paper proposes a semantic segmentation method that integrates hyperspectral imaging and deep learning. Through manifold learning for dimensionality reduction, multi-level feature fusion, and optimization with a dynamic sparse attention mechanism, precise identification of mold regions against complex backgrounds is achieved. By reducing high-dimensional spectral data to a three-dimensional space, the redundancy issue of hyperspectral data is effectively addressed, preserving the spectral difference characteristics between different molds and the background while providing a suitable input format for subsequent deep learning models. The designed CNN-Transformer hybrid network fully integrates the local detail capturing capability of CNNs with the global context modeling advantage of Transformers. Experiments demonstrate that the model achieves mean Intersection over Union (mIoU) scores of 90.1%, 96.8%, and 89.3% on datasets of Aspergillus niger, Penicillium citrinum, and Cladosporium cladosporioides, respectively, representing significant improvements over single CNN or Transformer models, especially in the recognition of weak and irregularly shaped molds. The introduced TOP-K dynamic sparse attention module screens key feature regions through channel and spatial variance, effectively suppressing background noise interference and reducing the false detection rate in complex backgrounds by approximately 3.5%. This study provides efficient technical support for the early monitoring and preventive conservation of mold on artifacts and can be applied to scenarios such as environmental monitoring in artifact storage facilities and pre-restoration damage assessment. However, the current method still has limitations: firstly, the model’s generalization ability in areas heavily covered with ink needs to be verified; secondly, the training data is concentrated on laboratory-cultured samples, and multi-source data from real artifact scenarios needs to be further expanded to improve robustness.