Non-Targeted Screening Method for Detecting Temporal Shifts in Spectral Patterns of Methicillin-Resistant Staphylococcus aureus and Post Hoc Description of Peak Features

Abstract

Non-targeted methods (NTMs) using matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI-TOF MS) show promise in bacterial resistance detection, yet temporal variations in spectral features pose significant challenges. These proteomic patterns, which characterize bacterial phenotypes and pathological functions, may vary over time due to bacterial adaptation, virulence, or resistance mechanisms, resulting in large prediction uncertainties and potentially degrading NTM performance. We present a comprehensive screening method to detect temporal changes in MALDI-TOF spectral patterns, demonstrated using methicillin-resistant and -susceptible Staphylococcus aureus (MRSA/MSSA) isolates collected over several years. Our approach combines convolutional neural networks (CNNs) with statistical methods, including significance testing, kernel density estimation, and receiver operating characteristics for dataset shift detection. We employ Gradient-weighted Class Activation Mapping (Grad-CAM) for post hoc feature description, enabling biochemical characterization of temporal changes. This analysis reveals crucial insights into the dynamic relationship between spectral data patterns over time, addressing key challenges in developing robust NTMs for routine applications.

1. Introduction

Non-targeted methods (NTMs) using matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI TOF MS) are a promising set of methods that can be used for bacterial species identification [1], sub-species or strain differentiation [2], and metabolic profiling [3], among others. By measuring the mass-to-charge ratio of ions produced from bacterial samples, such methods provide proteomic fingerprints in the form of spectra, which can be used (a) to compare a query against a database of reference spectra or (b) for supervised learning techniques, the latter of which we exploit here to detect changes in spectral patterns of antibiotic-resistant bacterial strains. Throughout this manuscript, we use NTMs to denote the complete analytical workflow, i.e., from sample preparation through spectral acquisition to machine learning-based classification [4].

The use of machine learning methods with MALDI TOF spectra has been a “well-trodden path” for some time now [5,6]. These computational methods excel at learning intricate spectral patterns that reflect biochemical variations associated with antibiotic resistance, offering promising NTMs for distinguishing methicillin-resistant Staphylococcus aureus (MRSA) from its susceptible counterpart (MSSA) [6,7]. Recent multicenter studies demonstrate rapid MRSA screening capabilities with performance comparable to conventional methods [8], whilst recent reviews highlight the importance of MALDI-TOF MS for both species identification and resistance profiling in veterinary contexts, particularly for bovine mastitis surveillance [9]. Advanced approaches incorporating uncertainty estimation further enhance the reliability of resistance predictions [10]. This enhanced analytical capability could significantly improve diagnostic precision, ultimately enabling more informed decision-making.

The current literature on studying antimicrobial resistance (AMR) using MALDI TOF is very broad and deep, but with conflicting evidence [11]. Recent comprehensive reviews emphasize that MALDI-TOF MS plays a crucial role in AMR surveillance across the food production chain, from livestock to food products [12,13,14], whilst highlighting challenges of dataset shift in clinical machine learning applications [12]. While several groups have demonstrated promising results in MRSA/MSSA discrimination [7,15,16,17], other investigations highlight insufficiency and uncertainty in this approach as a rapid diagnostic method [18,19,20]. These disparate outcomes likely stem from the inherent complexity of spectral data collection and analysis. Despite standardized sample preparation and analytical measurement methods (e.g., instrument configurations), spectra acquired from different (a) locations and (b) time points frequently pose significant challenges for integration into a unified predictive model. Focusing on the latter, one of the main practical challenges of NTMs is that the peak features of the spectra may change over time due to various factors such as genetic mutations [21], environmental conditions [22], experimental protocols, or instrument settings [19]. Recent validation studies confirm that ML models for AMR prediction experience significant performance degradation over time, with accuracy declining by 10–25% within 18 months without retraining [23]. These changes may affect the performance and reliability of NTMs for bacterial identification and AMR detection, as well as the interpretation and comparison of the results across different studies and laboratories [7,24]. Most recently, this topic of data heterogeneity has been studied by Park and colleagues, where they report the effects of MRSA-MSSA class imbalance and the effects of different sample preparation and laboratory environments for measurements [25]. This highlights the need to understand and account for the temporal changes in spectral peak features and understand how they impact method performance.

Temporal changes in MALDI-TOF data manifest through multiple, interrelated mechanisms. Studies often employ potentially confusing terminology such as (a) concept drift, (b) covariate shift, and (c) feature shift [26,27]. In practice, an example for (a) where new resistance patterns emerge or bacterial subgroups form; (b) when hospital-trained models face veterinary samples; and (c) where specific spectral patterns lose their discriminatory power. These terms, while theoretically distinct, often create confusion as their practical manifestations frequently overlap [28,29]. Rather than formally categorizing these phenomena, this work examines their collective impact through three interconnected aspects (see Figure 1A). First, the emergence of new resistance patterns could introduce novel spectral signatures, potentially compromising existing classification models. Second, shifts in bacterial population structure, where different resistance patterns becoming more prevalent over time, might affect model performance through changing class distributions. Third, specific peaks of interest (corresponding to specific protein complexes) may exhibit drift or instability, due to either instrumental factors or genuine biological changes in the bacterial proteome. Understanding these aspects is crucial for long-term implementation, as changes in any dimension could impact NTM reliability.

Through analysis of MALDI-TOF spectral data from MRSA and MSSA isolates collected over several years, this paper addresses the challenge of temporal variation in spectral features. Our proposed method comprises three main steps, each targeting the key questions of resistance patterns, population changes, and spectral stability. First, we detect temporal changes using convolutional neural networks (CNNs) trained on the complete dataset over the entire time span (Figure 1B), followed by a sequential approach to assess the significance of temporal differences in decision scores. Second, we conduct period-specific CNN training and classification to observe changes within MRSA. These shifts may not directly influence MRSA vs. MSSA classification but are essential for understanding intra-MRSA variations. By independently classifying different periods, we aim to further uncover changes in spectral patterns and target labels. The second step helps to detect the dataset shifts using the decision scores (Figure 1C). Here we present the use of statistical procedures like visualizing kernel density estimation (KDE) distributions and receiver operating characteristic (ROC) curve (Figure 1D). Lastly, we describe which set of peaks contribute to temporal differences, using post hoc feature description. Here, Gradient-weighted Class Activation Mapping (Grad-CAM) [30] is employed to identify relevant m/z ranges for each year (Figure 1E). Extracting suitable peak features that contribute to the discrimination of a spectrum is not only important for the interpretability and performance of the models but also helps detect biochemical changes. Post hoc feature description helps find distinctive patterns and features that lead to a classification decision. Ultimately, this comprehensive analysis enhances our understanding of the dynamic relationship between MRSA and MSSA across various years.

2. Materials and Methods

2.1. Spectral Dataset

MALDI-TOF MS data were collected between 2008 and 2021 at the Bundesamt fuer Verbraucherschutz und Lebensmittelsicherheit using a Bruker Microflex LT instrument. The dataset comprises spectra from 440 isolates of Staphylococcus aureus. Sample preparation followed standard protocols [15,31]. Briefly, bacteria were cultured on Columbia sheep blood agar plates for 18–24 h at 33–37 °C, transferred using a sterile loop to the MALDI target plate, overlaid with α-cyano-4-hydroxycinnamic acid matrix solution, and air-dried. Spectra were acquired in the mass range of 2000–20,000 Da. Using custom scripts, the raw spectra underwent baseline correction using a rolling median filter. Methicillin resistance status was confirmed genotypically by detection of the mecA gene using polymerase chain reaction (PCR). Isolates positive for mecA were classified as MRSA; mecA-negative isolates were classified as MSSA.

2.2. Convolutional Neural Network (CNN) Model Training and Decision Scores

Two sets of CNN model training were performed—(i) to classify MRSA and MSSA and (ii) to classify time periods. For both, the CNN architecture comprised three convolutional layers interspersed with max-pooling layers, implemented via Keras and TensorFlow [32,33]. To address class imbalance across years, weighted training using Keras’s class weights function was applied, with weights inversely proportional to class frequencies, meaning that years and classes with fewer spectra have higher weights and vice versa. This way, the model gives more importance to the underrepresented years and classes and learns to generalize better across the data.

The nested cross-validation (NCV) approach utilized a calibration set consisting of MRSA and MSSA spectra in duplicates [34,35]. For each internal validation fold, the calibration set was divided into training and testing subsets. Adhering to the NCV framework, CNN models were trained on the training subset and validated with the test subset. Cross-validation splitting was performed at the isolate level (analogous to GroupKFold), ensuring all duplicate measurements from any given isolate remained in the same fold. This prevents data leakage between training and testing sets and ensures valid estimation of between-isolate and within-isolate variance components. The CNN employed gradient descent optimization to minimize the loss function by iteratively adjusting the learnable parameters through backpropagation until convergence at each layer. The output from the CNN comprised a standardized logit-transformed probability value which was used for decision score computation [34]. The classification decision hinged on a threshold, set to zero in this context, such that positive decision scores indicated class A, while negative scores indicated class B.

2.3. Performance Characteristics

Dual approaches were employed to evaluate the classifier’s performance. First, a confusion matrix quantified classification outcomes through true/false positives (TPs/FPs) and negatives (TNs/FNs). Second, the decision scores were analyzed to decompose variance components, providing deeper insights into classification reliability.

For binary non-targeted methods, the evaluation framework integrated both discrete classification metrics and continuous decision scores. Following Uhlig et al. [36,37], let $Z_{ij}$ denote the decision score for isolate $i$, spectrum $j$. The model for classification based on quantitative scores is as follows:

$$Z_{ij}={\mu }_{r\left(i\right)}+{\alpha }_i+{\beta }_{ij}$$

(1)

$$\mathrm{where} r\left(i\right)=\left\{\begin{array}{cc}1& \widehat{=}i \mathrm{b}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{n}\mathrm{g}\mathrm{s} \mathrm{t}\mathrm{o} \mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e} \mathrm{A}\\ -1& \widehat{=}i \mathrm{b}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{n}\mathrm{g}\mathrm{s} \mathrm{t}\mathrm{o} \mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e} \mathrm{B}\end{array}\right.$$

(2)

Here, ${\mu }_{r\left(i\right)}$ represents the mean score for respective classes A and B. ${\alpha }_i$ captures the isolate $i$ specific (random) variation and ${\beta }_{ij}$ accounts for the (random) deviation of the decision score for spectrum $j$ of isolate $i$. Assuming homogeneous variance within classes, two-way random effects analysis of variance ANOVA was used to estimate variance components.

$$\begin{gathered} {Var(\alpha }_i;i\in A)={\sigma }_{1,A}^2 and {Var(\alpha }_i;i\in B)={\sigma }_{1,B}^2 \\ Var\left({\beta }_{ij};i\in A\right)={\sigma }_{2,A}^2 and Var({\beta }_{ij};i\in B)={\sigma }_{2,B}^2 \end{gathered}$$

(3)

Based on the above variance components, the classification variances were calculated as follows:

$$\begin{gathered} {\sigma }_{classification,A}^2={\sigma }_{1,A}^2+{\sigma }_{2,A}^2 \\ {\sigma }_{classification,B}^2={\sigma }_{1,B}^2+{\sigma }_{2,B}^2 \end{gathered}$$

(4)

The smaller the classification variances, the better the classification power. The discrimination power $\mathcal{D}\mathcal{P}$ of the NTM, derived from the concept of Fisher’s discrimination index, was then calculated as:

$$\mathrm{Discrimination} \mathrm{power} (\mathcal{D}\mathcal{P}) ={\displaystyle \frac{{\mu }_1-{\mu }_{-1}}{{\sigma }_{classification, A}+{\sigma }_{classification,B}}}$$

(5)

${\sigma }_{classification}$ in the denominator were replaced by estimated standard deviation values ${\mathrm{s}}_{classification}$ evaluated from the decision scores. Calculation of variance components was performed according to ISO 5725-2 [38], with PROLab Plus v2025.11.6.0 statistical software (QuoData GmbH, Germany). $\mathcal{D}\mathcal{P}$ above 2 means good discrimination between the classes. If $\mathcal{D}\mathcal{P}$ is between 1 and 2, then reasonable discrimination is possible, and if $\mathcal{D}\mathcal{P}$ is less than 1, then there is poor discrimination.

2.4. Significance Testing and t-Values

The sequential t-test analysis was performed on the decision scores to assess significant differences between time periods $\left(\tau \right)$. At each sequential step, two time periods, $\tau 1$ and $\tau 2$, were compared. If no significant differences were found, the periods were combined for the subsequent steps.

In case of unequal variance of decision scores, t-values for the two-time spans, $\tau 1$ and $\tau 2$, were calculated as a ratio of difference in the means ($\widehat{\mu }$) divided by pooled variance.

$$t={\displaystyle \frac{\widehat{{\mu }_{\tau 2}}-\widehat{{\mu }_{\tau 1}}}{\sqrt{\frac{s_{\tau 2}^2}{n_{\tau 2}}+\frac{s_{\tau 1}^2}{n_{\tau 1}} }}}$$

(6)

ROC curves were plotted and the area under the curve (AUC) was evaluated. A large AUC value suggests that the two time periods can be very well distinguished, with a value close to 1 signifying near perfect discrimination.

2.5. Feature Extraction Using Grad-CAM

Grad-CAM is a feature extraction and explainability technique based on neural network dissection, which aims to visualize and helps to interpret the internal representations learned by deep networks [30]. It is based on the idea of using the gradients of the output class with respect to the feature maps of the last convolutional layer of the model to produce a coarse localization map of the input [39]. Feature maps are the output of a convolutional layer in a neural network, which consists of a set of activation values for each filter (or kernel) applied to the input [40]. Figure 1E shows how Grad-CAM evaluation involved four key steps: First, the prediction output was obtained from the CNN in the form of the decision score for the class of interest (e.g., MRSA). Next, the gradient of the score $S_R$ with respect to the feature map $A_{convolution layer. l}$ of the last convolutional layer was evaluated. This was followed by performing a global average pooling on the gradients to obtain the weights ${\alpha }_R^k$ for each feature map. These weights represented the importance of each feature map for the MRSA. Each feature map $A_{convolution layer. l}$ was then multiplied by its corresponding weight ${\alpha }_R^k$ and summed to obtain the Grad-CAM activation map $L_R$. This map has the same dimension as that of the feature map $A_{convolution layer. l}$. Lastly, the Grad-CAM activation map $L_R$ is resized to match the size of the input spectrum and overlaid on the original spectrum with a color map. The resulting spectrum shows the regions that the CNN focuses on when predicting the MRSA class.

2.6. Peak Features Changing over Time

After having identified the peak features, to identify the specific peak features that change over the years, we performed a one-way ANOVA analysis, using the quantitative Grad-CAM intensities. We calculated the F-value for the features, which can be used to determine the features that contribute most to the classification. By performing an ANOVA analysis on the Grad-CAM feature values, we can test whether there are significant differences among the mean values of each peak for each year. These peaks are the ones that have changed significantly over the years and can be considered potential biomarkers or indicators of temporal variation.

3. Results

3.1. Identifying Temporal Changes Using Classification Decision Scores for MRSA and MSSA

To address the key objective of identifying spectral shifts, first the CNNs were trained using all available spectral data for MRSA and MSSA, i.e., from 2008 to 2021, employing the 5-fold NCV technique. Samples from each year were evenly distributed across all folds to ensure an equal representation of years. Internal validation decision scores were evaluated and no spectra for external validation were used as the aim was not to develop an NTM to predict MRSA or MSSA, but rather to gain insights into the varying spectral patterns of MRSA and MSSA that affect their classification. In doing so, one can detect if there are changes in the population with different spectral patterns.

Figure 2 shows a Youden plot with decision scores for distinguishing MRSA and MSSA. Each point on the plot represents an isolate, with decision scores for duplicate measurements on the two axes. Considering a decision threshold of 0, a confusion matrix can be constructed, resulting in 14 false positives and 14 false negatives, resulting in a true positive rate (sensitivity) of 91% and a false positive rate (1—specificity) of about 3.3%. From the Youden plot, two direct observations emerge. First, for MSSA samples (spanning from 2008 to 2017), the classification remains predominantly accurate, i.e., the majority of the points lie in the third quadrant. And secondly, the decision scores of replicate samples display considerable variability, as evidenced by their deviation from the line of identity.

To examine the presence of “sub-populations” within MRSA, the point clouds are drawn in different shapes by known year label. MRSA samples from 2008 to 2015 (red squares) and 2021 (red upper triangles) are farthest from the MSSA of 2008 to 2017 (blue circles), which implies that they can be more distinctly classified. The point cloud distribution additionally indicates that MRSA samples from 2017 (red diamonds) and 2019 (red lower triangles) are notably close to the MSSA samples. This suggests that clear classification between MRSA samples from these years is challenging. There is also an indication that 2019 is similar to 2017, but 2021 is different. However, further differentiation is not possible, i.e., comparing 2021 with other years, because of the fewer available data points.

Such a result prompts the question of how one can identify evolving patterns or spot temporal variations within the data over time. In order to identify whether differences in subgroups are significant, we performed a sequential t-test procedure.

3.2. Sequential Significance Testing of Decision Scores

Following the observations from the Youden plot, sequential significance testing to systematically identify temporal groupings was performed. Figure 3 illustrates the step-by-step approach. The sequential testing began with step 1, comparing two pairs of years with minimal data points: 2019 versus 2021 (denoted by red 1) revealed statistically significant differences, while 2008 versus 2009 (marked as green 2) showed no significant differences. Following this initial analysis, 2008 and 2009 were combined for subsequent comparisons. Throughout the figure, green-shaded boxes indicate pairs of periods where decision scores show no statistical differences, while red-shaded boxes highlight periods with statistically significant differences.

Step 2 revealed that decision scores from 2011 and 2013 showed no significant differences. Proceeding to step 3, the comparison of the combined 2011 + 2013 period with 2015 also showed no statistical differences, enabling the formation of larger temporal groups. Further analysis demonstrated that decision scores from 2017 and 2019 were statistically similar, while the data from 2008 to 2015 maintained statistical consistency throughout the analysis.

The sequential testing culminated in step 6, which definitively identified three distinct temporal periods: 2008–2015, 2017–2019, and 2021. Final statistical tests confirmed significant differences between these three periods. The sequential analysis provides strong evidence that MRSA isolates from 2017 and 2019 exhibit similar characteristics but differ significantly from other periods. While this methodology effectively addresses the presence of changes in spectral patterns, it is important to note that these statistical differences cannot be directly interpreted as qualitative differences in MRSA isolate behavior.

3.3. Finding Breaks by Year-Wise Training

The sequential significance testing helped identify three time periods from the decision scores. To further validate this result, several CNNs were trained to perform classification for two time periods separately for MRSA and MSSA. Figure 4 shows the (A) rug plot charts for decision scores, showing the distribution of scores and (B) ROC curves showing discriminatory relation of the model through the lens of false-positive rate (FPR) and true-positive rate (TPR). Calculating the t-values using Equation (6), we obtain large t-values when comparing MRSA for periods (a) 2015 versus 2017 (value of 6.9) and (b) 2017 versus 2019 + 2021 (value of 4.67). Considering a critical t-value of 2, one can consider the difference in the decision scores to be significant for the above two periods. The ROC curves also show high discrimination for these two periods with AUC values of 0.9 and 0.83, respectively. For the other periods, the AUC value is around 0.5.

Likewise, MSSA for the period 2015 versus 2017 also shows some evidence of differences in decision scores with a relatively high t-value of 3.69 and an AUC of 0.75. This finding is particularly intriguing as antibiotic resistance-associated proteins should theoretically not influence MSSA spectra. Several potential explanations warrant further investigation: the presence of borderline resistance cases that are still classified as MSSA, regional and temporal variations in MSSA subtypes across different geographical regions, differences in measurement conditions during spectral acquisition, or variations in isolate processing protocols that could affect bacterial viability and proteome expression.

The combination of t-values and ROC-AUC metrics provides complementary evidence: t-values establish statistical significance of temporal differences, while ROC-AUC values quantify classification performance. These metrics jointly support the temporal breaks identified in the previous section.

3.4. Tracking Features over Time

3.4.1. Feature Extraction Heatmaps

We proceeded to neural network dissection with Grad-CAM to address the question of if there are changes in specific peaks. The left panels of Figure 5A show the baseline-corrected spectra (black line) overlaid with the Grad-CAM heatmaps presenting one representative spectrum from each year. These heatmaps quantitatively illustrate the contribution of each spectral region to the model’s predictions, where warmer colors indicate stronger contributory features. For instance, in the spectrum in the first row, multiple specific bands demonstrate high influence on the classification. Notably, across all spectra, regions corresponding to m/z 2000–4000 consistently show high importance, indicated by prominent dark red bands. Several studies also report the presence of discriminatory peaks in that range [6,11,41].

By bringing together the Grad-CAM results for all the spectra, a consolidated heatmap is constructed in Figure 5B. In this heatmap, each row of pixels corresponds to a spectrum and each column corresponds to a mass-by-charge ratio. Once again, the color intensity represents the classification importance derived from Grad-CAM analysis, with warmer colors indicating higher significance. The heatmap reveals distinct patterns of CNN focus across different years, suggesting temporal variations in MRSA spectra.

Overall, Grad-CAM demonstrates its utility as a sophisticated visualization technique for interpreting CNN classification behavior, offering insights into the network’s learned features and their relationship to output classes across varying inputs. It can also be used for development and optimization of NTMs. The heatmap primarily provides a qualitative or global perspective of the network-learned spectral features, and detailed year-to-year comparisons may only be partially feasible through visual inspection.

3.4.2. Detecting Specific Peaks Sensitive to Temporal Changes

To quantitatively evaluate the year-to-year changes in the Grad-CAM features, one-way ANOVA analysis is performed for the Grad-CAM feature value with the year. High F-values mean that the peak regions show large variation in importance between the years. Six exemplary spectral regions are taken with relatively high F-values and plotted in Figure 6. The overlap of features learned by the models with recognized resistance patterns reported in the literature is a promising result. Further work is warranted here in terms of biochemical characterization of the proteins responsible for the peaks.

3.5. Classification Performance Based on Different Time Periods

Following the identification of temporal breaks in 2017, we investigated classification performance by analyzing pre-2017 and post-2017 periods separately. Figure 7 presents Youden plots for MRSA classification across these distinct periods: (A) 2008–2015 and (B) 2017–2019. The 2008–2015 period demonstrated a TPR, or sensitivity, of 99% and a very low FPR (<<1%), i.e., very high specificity. This represents a substantial improvement over the non-temporally segregated analysis presented in Section 3.1. The repeatability standard deviations for MRSA and MSSA are 0.404 and 0.285, respectively, with classification standard deviations of 0.419 and 0.308, respectively. Using Equation (5), $\mathcal{D}\mathcal{P}$ is calculated to be 2.75. Recollect that values above 2 mean very good discrimination between the classes. The 2017–2019 period showed markedly different characteristics, with sensitivity decreasing to 52% (FPR of 11%). Furthermore, it is also clear that the variation between the replicates has increased (spread of the red and blue points), with repeatability standard deviations for MRSA and MSSA of 0.815 and 0.729 and classification standard deviations of 0.815 and 0.821. The resulting $\mathcal{D}\mathcal{P}$ of 1.22 indicates that the performance characteristics for the NTM for identifying MRSA from MSSA have significantly deteriorated, necessitating a revision and update of the NTM [4]. This observation aligns with recent validation studies demonstrating similar temporal performance degradation in MALDI-TOF-based AMR prediction models [23].

4. Discussion

Proteomic patterns, such as protein expression, modification, and interaction, are widely used to characterize the biological and pathological functions of bacteria. However, these patterns may vary over time due to various processes, such as bacterial growth, adaptation, virulence, or resistance. In this paper, we present a study of the temporal changes in proteomic patterns of MRSA and propose a screening method. We use a large sample of MALDI TOF data collected over several years. We propose and apply statistical methods and machine learning techniques to find suitable features and track their changes over the years. We also discuss the implications of our results and how the identified peaks can help with biochemical characterization as well as the prospects for future studies with emerging technologies.

We use CNNs to learn cross-year feature representations that can capture the biochemical variations associated with antibiotic resistance or susceptibility. We evaluate our method on a large dataset of mass spectrometry data collected from 2008 to 2021. Quantitative decision scores from CNNs were evaluated and used not only in performance characterization but also to evaluate the changes over the years. Using decision scores, variance components are evaluated from which the precision parameters are calculated such as repeatability standard deviation and classification standard deviation, which can help to describe the distribution of points and their farness from each other. Detecting shifts in data over years using quantitative decision scores is an important novelty of this work, complementing recent approaches employing unsupervised information geometric projections for early identification of dataset shift patterns [42]. Such quantitative monitoring approaches would not have been possible merely with qualitative binary outcomes. Feature descriptions are primarily important for three main reasons. First, they ensure that the method works “as expected”; feature extraction can help one understand if the model identifies features that are “logical” or “relevant”, or if the model relies on arbitrary patterns in the data to make a classification. Second, they help one understand the reason for the result, as one can understand why the model classified the sample as MRSA, and, more importantly, one can determine whether the features have changed over the years. Lastly, they help develop targeted methods or improved NTMs. Neural network dissection methods like Grad-CAM are a type of post hoc feature description method that analyze the activations of the hidden layers of a network and try to find meaningful associations between the units in those layers and the input features or the output classes. We apply the Grad-CAM method to detect the relevant mass ranges for each year and compare them with the temporal breaks detected by statistical methods. We show that our approach can provide useful insights for biochemical analysis, as well as potentially accounting for the possible confounding factors of sample selection, isolate preparation, and measurement circumstances.

Our methodological approach differs from classical MALDI-TOF AMR studies in several fundamental aspects. First, whilst most investigations focus on cross-sectional discrimination at single time points [8,9,10], we explicitly address temporal validation, evaluating whether models remain valid as bacterial populations and resistance patterns evolve. This temporal perspective is essential for clinical deployment but remains rare in the literature [33,39]. Second, unlike approaches seeking singular biomarker peaks indicative of specific resistance mechanisms [31], our CNN-based pattern recognition captures complex multi-peak signatures that may prove more robust to subtle instrumental drift whilst remaining interpretable through Grad-CAM analysis. Third, our quantitative decision score framework enables rigorous variance component decomposition and significance testing impossible with binary classification outcomes alone [27,28]. This statistical foundation allows us to distinguish genuine biological/epidemiological shifts from measurement variability, a critical capability for antimicrobial surveillance programs where false-positive drift detection could trigger unnecessary model retraining.

Traditional accuracy-based performance metrics (where accuracy = (TP + TN)/(TP + FP + TN + FN)) might be misleading when dealing with extremely imbalanced data where one class considerably dominates the other(s). They are commonly reported in the literature where new NTMs are described. The model in the NTM may simply predict the majority class and have a large accuracy value. Here there is a risk of having a false sense of superior NTM performance. It is advisable to exercise caution or, ideally, refrain from employing accuracy metrics commonly applied for model evaluation when faced with imbalanced class structures, as their interpretation may yield misleading insights. Quantitative decision scores should instead be used, as performed here, from which sensitivity and specificity can be evaluated.

The work herein is limited by all means to a screening method and does not offer any confirmatory procedures because so far it does not provide any information on the biochemical nature of the changed patterns or extracted features. However, the integration of such screening approaches into decision support systems has shown promise for accelerating antimicrobial stewardship [43]. Nevertheless, the approaches offer a pragmatic and powerful approach that can be implemented via the use of simple tools as shown in this work. Therefore, further biochemical analysis is needed to identify and describe the proteins responsible for the peaks. The core reasons for the changes need further investigation. Here, another possible reason is the different sampling processes of the isolates, which can also affect how the spectra look. Overall, it is clear that with such a screening method, one can easily detect changes in data and generate several hypotheses to then confirm.

5. Conclusions

Data shifts and change in the samples over time is a huge challenge for any NTM for species typing and AMR programs, let alone other analytical sciences. And this work helps to provide tools for handling such scenarios. In this paper, we propose a non-targeted screening method to detect the changes in peak features in mass spectrometry data over the years. We have shown that by identifying the temporal breaks, strategic dataset selection, and model adaptation, one can maintain classification accuracy across different time periods.

Performance evaluation of NTMs is very important not only to trust the results, but also for gaining wider acceptance in routine use. We show how performance evaluation can be performed using the underlying decision scores obtained from the CNN models. In this work, several data experiments were conducted with MRSA and MSSA MALDI TOF spectra. To investigate the impact of changing samples, different combinations of available data were put together to test out hypotheses. The endeavor of NTM development encounters heightened challenges when the MRSA (or any class of interest) contends with a limited pool of unique available samples, compounded by evolutionary changes in features across temporal dimensions. This is something we expect in reality and hence the work in this report describes considerations and options to tackle these challenges. The methodological aspects presented here, sequential significance testing of decision scores, period-specific model training for drift detection, and Grad-CAM-based feature tracking, provide a generalizable framework for the surveillance of temporal changes in bacterial populations. This is particularly relevant for AMR monitoring programs, where spectral pattern shifts may indicate emergence of novel resistance mechanisms, clonal replacements, or methodological drift. Early detection of such changes enables timely model updating, maintaining diagnostic accuracy for antibiotic stewardship. The approaches are readily applicable beyond S. aureus to other priority pathogens where MALDI-TOF-based AMR prediction is being developed. Lastly, the approaches described, when provided in a suitable environment, add to the toolbox for developing superior methods of measurement and validating them. Given the documented performance degradation of ML models over time [23], future work should focus on implementing the screening method described as an automated method for detecting spectral drift and implementing real-time model updates to ensure reliable clinical diagnostics. This aligns with broader initiatives in medical microbiology to establish standardized protocols and interdisciplinary collaboration frameworks for AI implementation [44].