A Focus on Important Samples for Out-of-Distribution Detection

Abstract

To ensure the reliability and security of machine learning classification models when deployed in the open world, it is crucial that these models can detect out-of-distribution (OOD) data that exhibits semantic shifts from the in-distribution (ID) data used during training. This necessity has spurred extensive research on OOD detection. Previous methods required a large amount of finely labeled OOD data for model training, which is costly or performed poorly in open-world scenarios. To address these limitations, we propose a novel method named focus on important samples (FIS) in this paper. FIS leverages model-predicted OOD scores to identify and focus on important samples that are more beneficial for model training. By learning from these important samples, our method aims to achieve reliable OOD detection performance while reducing training costs and the risk of overfitting training data, thereby enabling the model to better distinguish between ID and OOD data. Extensive experiments across diverse OOD detection scenarios demonstrate that FIS achieves superior performance compared to existing approaches, highlighting its robust and efficient OOD detection performance in practical applications.

1. Introduction

When machine learning models are deployed in open-world environments, they often encounter inputs from out-of-distribution (OOD) data, which exhibit semantic shifts from id-distribution (ID) data used during training [1,2,3]. It is imperative that models avoid making predictions on OOD data, especially in safety-critical domains [4,5], such as autonomous driving and medical diagnosis [6]. In these fields, making arbitrary predictions on OOD data can lead to catastrophic consequences. This concern has spurred significant research on OOD detection in the field of reliable machine learning in recent years [7,8,9,10,11,12].

OOD detection involves two main tasks: accurately predicting ID data and effectively detecting OOD data [10,13,14]. Existing methods can be broadly categorized into two types based on whether they introduce additional training data [15]. The first type trains a model using only ID data to achieve accurate classification and then employs post-processing methods based on model outputs to distinguish between ID and OOD data [16,17,18,19]. For example, [13,20] design scoring functions that better capture the distinct patterns of ID and OOD data, [15,18] use different distance metrics to differentiate ID and OOD data based on features extracted by the model, and [13,21,22] rely on gradient-based information to make this distinction. However, due to the lack of explicit OOD knowledge, these methods, despite theoretical guarantees, often fall short of achieving powerful OOD detection performance.

The second type of method uses additional auxiliary data to enhance the ability of the pre-trained model to detect OOD data. For instance, refs. [6,23] design various OOD data generation methods to create a diverse set of OOD data for model training, while others expand the coverage of OOD data through different methods to improve the ability of the model to detect OOD data [1,24]. These methods generally outperform the first category in OOD detection. However, these methods trend focus on acquiring more OOD data for training, with little attention paid to fully leveraging the potential of existing OOD data. Motivated by this gap, in this paper, we analyze the usefulness of different data for model training and propose a method that efficiently utilizes existing data to train the model.

In Section 3, we empirically demonstrate through experiments the significant variations in the impacts of different data samples on model training. Unlike classical active learning paradigms [25], which typically prioritize more challenging samples for training, our findings reveal that OOD data correctly classified by a pre-trained model are more beneficial for OOD detection training. Conversely, more difficult OOD samples are found to exert a negative influence on model training.

The samples that the model can confidently and correctly classify, which are referred to as easily distinguishable samples, play a pivotal role during fine-tuning. Consequently, a principled approach for OOD fine-tuning involves preferentially selecting such readily classifiable samples for training. Implementing this strategy necessitates developing a rigorous methodology to systematically assess and identify samples that exhibit higher distinguishability to the model.

Fortunately, prior research conducted by [1,26] has shown that pre-trained models exhibit inherent capabilities for OOD detection to a certain extent. Building on this insight, we can use the OOD scores predicted by pre-trained models to identify samples that it can correctly recognize samples based on a certain threshold. By fine-tuning the model using these important samples, we can enhance its OOD detection ability.

In Section 4, we design an efficient training algorithm named focus on important samples (FIS) that iteratively performs prediction and training, as illustrated in Figure 1. This iterative process enables the model to learn from key samples that are more conducive to training. Ultimately, this approach not only ensures accurate classification of ID data but also maximizes its capability to effectively identify OOD data.

We conducted comprehensive and rigorous experiments across a variety of OOD detection scenarios. Consistently across all experimental settings, our method achieved superior OOD detection performance by using only a subset of the data, outperforming methods that utilized the entire dataset. This demonstrates the reliability of our proposed approach in practical applications. Furthermore, we performed extensive ablation studies, which confirmed the robustness of our method.

We summarize the main contributions of this paper as follows:

• We empirically analyzed the varying impacts of different data on model training through comparative experiments, demonstrating that samples that the model can correctly identify play a more critical role in the OOD detection fine-tuning stage.
• We proposed a highly efficient and targeted method for utilizing training data, which selects samples that are more beneficial to model training based on the OOD scores predicted by the model during the training process.
• Through extensive experiments, we validated the efficiency, reliability, and robustness of our proposed method, FIS, in OOD detection training.

The organization of this paper is as follows: in Section 2, we introduce and explain the key concepts and notations relevant to this study. In Section 3, we analyze the varying impacts of different data on model performance during the training process. In Section 4, we propose the FIS algorithm, which efficiently leverages data to optimize training performance. In Section 5, we conduct extensive experiments to validate the reliability and robustness of our method. Finally, in the last section, we conclude this paper.

2. Preliminary

In this section, we provide a comprehensive introduction to the key concepts relevant to this paper and explain the key notations.

We consider the multi-class classification task. Let $\mathcal{X}$ and $\mathcal{Y}=\{1,\cdots ,c\}$ denote the input space and label space corresponding to the ID distribution, respectively. Then, denote ${\mathbb{P}}_{\mathrm{ID}}$ as the ID distribution defined over $\mathcal{X}\times \mathcal{Y}$, and ${\mathbb{P}}_{\mathrm{OOD}}$ is the OOD distribution. Let Y be a random variable from space $\mathcal{Y}$, i.e., $Y\in \mathcal{Y}$, and let $Y_O$ be a random variable from unknown OOD label space. We have $Y_O\notin \mathcal{Y}$, which means that the ID sample ${\mathbf{x}}_{\mathrm{ID}}$ and OOD sample ${\mathbf{x}}_{\mathrm{OOD}}$ have different label space [26]. Then, we can define the ID training data as follows:

$$D_{\mathrm{ID}}=\{({\mathbf{x}}_{\mathrm{ID}}^1,y^1),\cdots ,({\mathbf{x}}_{\mathrm{ID}}^n,y^n)\}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{\mathbb{P}}_{\mathrm{ID}}.$$

(1)

Furthermore, the goal of OOD detection can be defined as follows:

Definition 1

(OOD Detection [26]). OOD detection primarily accomplishes two tasks: (a) correctly classifying ID samples and (b) enabling the model to detect OOD samples that may be encountered during the inference phase.

These two goals are achieved through the design of different OOD scoring functions. We present the formal definition of the OOD scoring function as follows:

Definition 2

(OOD Scoring Function [1]). With a pre-trained classifier $\mathbf{f}$, the OOD scoring function $s(·;\mathbf{f})$ maps the input $\mathbf{x}$ to a value, i.e., $s(;\mathbf{f}):\mathcal{X}\mapsto \mathbb{R}$, such that the input data can be classified as ID or OOD data via a level-set estimation, i.e.,

$$h\left(\mathbf{x}\right)=\left\{\begin{array}{cc}\mathit{ID},\hfill & \mathit{if}s(\mathbf{x};\mathbf{f})\ge \lambda ,\hfill \\ \mathit{OOD},\hfill & \mathit{otherwise},\hfill \end{array}\right.$$

(2)

where $h\left(\mathbf{x}\right)$ is the binary decision between ID and OOD, and λ is a pre-defined parameter.

Existing approaches leverage pre-trained classifiers and utilize the information from logit outputs [10,13,16,17,27,28,29], embedding features [15,18,30,31,32,33,34], or gradient information [13,21,22] to devise various OOD scoring functions. Recent research has focused on employing non-parametric methods to design efficient scoring functions [18]. To facilitate better understanding and ease the exposition in subsequent sections, we provide two commonly used OOD scoring functions below.

Maximum Softmax Prediction (MSP): This is a widely known baseline proposed by [10] that employs the maximum value from the softmax output distribution as an indicator for OOD detection, with the scoring function provided in the following:

$$s_{\mathrm{MSP}}(\mathbf{x};\mathbf{w})=\underset{j}{max}{\mathtt{softmax}}_j\mathbf{f}(\mathbf{x};\mathit{\theta }),$$

(3)

where ${\mathtt{softmax}}_j(·)$ indicates the j-th softmax output. In an ideal scenario, OOD data should not correspond to any specific class in the predictions, resulting in uniformly low softmax values across all dimensions. However, in practice, MSP is prone to errors due to the widely known problem of overconfident predictions [10,16].

MaxLogit Prediction (MLP): Ref. [35] proposed an enhanced approach to MSP by utilizing maximal logits rather than softmax probabilities for constructing the scoring function, namely,

$$s_{\mathrm{MLP}}(\mathbf{x};\mathbf{w})=\underset{j}{max}{\mathbf{f}}^j(\mathbf{x};\mathit{\theta }),$$

(4)

where ${\mathbf{f}}^j(\mathbf{x};\mathit{\theta })$ represents the j-th dimension of the model output vector. Empirical results demonstrate that the MLP score outperforms MSP, particularly in scenarios with large ID label spaces, making it a prevalent choice in practical applications.

Building on these scoring functions, [36] proposed using additional auxiliary OOD data to fine-tune models, a method widely known as outlier exposure (OE). This innovative approach allows models to learn from OOD data and has demonstrated strong performance. Since then, numerous studies have been based on OE. Our method is also grounded in OE, and we provide the classic definition of OE below.

Outlier Exposure (OE): [36]. Given the ID training data $D_{\mathrm{ID}}$ and the additional auxiliary OOD data set $D_{\mathrm{AUX}}$, we defined the following:

$$D_{\mathrm{AUX}}=\{{\mathbf{x}}_{\mathrm{AUX}}^1,\cdots ,{\mathbf{x}}_{\mathrm{AUX}}^m\}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}.}{\sim }{\mathbb{P}}_{\mathrm{AUX}},$$

(5)

where ${\mathbb{P}}_{\mathrm{AUX}}$ is the auxiliary OOD distribution. We use ${\mathbb{P}}_{\mathrm{AUX}}$ instead of ${\mathbb{P}}_{\mathrm{OOD}}$ because the auxiliary OOD distribution ${\mathbb{P}}_{\mathrm{AUX}}$ may not perfectly match the true OOD distribution ${\mathbb{P}}_{\mathrm{OOD}}$ [24]. Then, OE aims to fine-tune the model to enhance its ability to detect OOD samples. Specifically, the training is processed by minimizing the following empirical risk:

$$\widehat{R}(\mathbf{f};\mathit{\theta })={\widehat{R}}_{\mathrm{ID}}(\mathbf{f};\mathit{\theta })+\alpha ·{\widehat{R}}_{\mathrm{OOD}}(\mathbf{f};\mathit{\theta }),$$

(6)

where ${\widehat{R}}_{\mathrm{ID}}$ improves the model ability to classify ID data, which can be written as follows:

$${\widehat{R}}_{\mathrm{ID}}(\mathbf{f};\mathit{\theta })={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\ell }_{\mathrm{ID}}\left(\mathbf{f}({\mathbf{x}}_{\mathrm{ID}}^i,y;\mathit{\theta })\right),$$

(7)

and ${\widehat{R}}_{\mathrm{OOD}}$ enhances the model’s ability to detect OOD data, which can be written as follows:

$${\widehat{R}}_{\mathrm{OOD}}(\mathbf{f};\mathit{\theta })={\displaystyle \frac{1}{m}}\sum _{i=1}^m{\ell }_{\mathrm{OOD}}\left(\mathbf{f}({\mathbf{x}}_{\mathrm{OOD}}^i;\mathit{\theta })\right).$$

(8)

OE-based methods allow models to be further fine-tuned, thereby exhibiting more powerful OOD detection performance [16,36,37]. Accordingly, various studies have designed more efficient fine-tuning approaches from different perspectives, such as sampling strategies [37,38,39], adversarial robust learning [15,38,40], meta-learning [41,42], and regularization strategies [43]. Some research has also explored various outlier synthesis strategies when OOD data are difficult to obtain [6,23,44]. Although outlier exposure often yields positive results, the discrepancy between auxiliary data and real OOD data largely limits their effectiveness in real-world detection. In recent studies, [1,24] investigated using data generation to expand the coverage of auxiliary OOD data to reduce the gap between distributions and achieve better performance. Unlike previous methods that aimed to obtain more data, our approach focuses on screening a small amount of data that are more important for OOD detection training, allowing the OOD data to fine-tune the model. Based on the above discussion, this paper primarily addresses the following problem.

Problem Setup: Previous methods either exhibit suboptimal performance or require a large amount of additional OOD data to assist model training, which can be highly resource-intensive, especially in large-scale complex systems or when dealing with massive datasets [39,45]. The primary challenge addressed in this paper is how to fully leverage available data to train the model efficiently. To achieve this, we need to tackle two key questions: first, which data are more important for model training? Second, how can we identify and locate these important data? We will address these two questions in the subsequent sections.

3. Data Importance Analysis

In this section, we conduct experiments using CIFAR-100 as ID data, Tiny-ImageNet-200 as auxiliary OOD data, and Textures as the test OOD dataset to systematically analyze which types of data are most beneficial for OOD detection training. The main results are summarized in Figure 2.

Given that the FPR95 metric is optimized by lower values, our experimental findings reveal an intriguing phenomenon: when training with more challenging OOD data, the training performance begins to deteriorate after several iterations. Conversely, when utilizing OOD data that the pre-trained model can correctly detect, the model converges more rapidly than when using all OOD data for training. The final results further indicate that training with the most difficult OOD data yields the poorest performance, whereas training exclusively with OOD data that the pre-trained model can correctly detect outperforms training with all auxiliary OOD data.

We conduct further experiments to fine-tune the model using $D_{\mathrm{c}}$ as auxiliary OOD data. Meanwhile, $D_{\mathrm{w}}$ is employed as OOD data for testing to evaluate the OOD detection performance. The selected dataset sizes are as follows: $D_{\mathrm{c}}$ contains 29,275 samples, and $D_{\mathrm{w}}$ contains 37,575 samples, resulting in a ratio of $D_{\mathrm{w}}$ to $D_{\mathrm{c}}$ of approximately 1.28:1. The results are presented in

Table 1. Comparison of the OOD detection performance when $D_{\mathrm{w}}$ serves as test OOD data. The results are expressed as percentages. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred.

Auxiliary OOD Data	FPR95 ↓	AUROC ↑	ID ACC. ↑	Time (s) ↓
$D_{\mathrm{all}}$	0.08	99.98	75.51	240.93
$D_{\mathrm{c}}$	0.08	99.98	75.71	108.48

. It can be observed that fine-tuning the model using the entire OOD dataset $D_{\mathrm{all}}$ yields nearly identical performance compared to fine-tuning using only $D_{\mathrm{c}}$. Moreover, the time required for fine-tuning with $D_{\mathrm{c}}$ is less than half of that required when using $D_{\mathrm{all}}$. It is important to note that $D_{\mathrm{all}}$ encompasses the $D_{\mathrm{w}}$ portion of the data, whereas $D_{\mathrm{c}}$ does not include $D_{\mathrm{w}}$.

We can infer from these observations that among the auxiliary OOD data, the samples that are more beneficial for model training are those relatively simple OOD instances that the pre-trained classifier can accurately detect. This is likely because these correctly detected OOD data possess more distinctive features compared to ID data, enabling the model to benefit from learning these patterns. In contrast, the OOD data that the model fails to detect exhibit lower distinguishability from ID data, thereby potentially impairing the ability of the model to learn to differentiate between ID and OOD data.

Therefore, it is sufficient to identify and utilize simpler OOD data that the pre-trained model can accurately distinguish for training purposes. We will subsequently propose a specific algorithm to achieve this.

4. Methodology

Based on the analysis presented in Section 3, we now introduce a specific training algorithm in this section. The detailed procedure of the algorithm is summarized in Algorithm 1.

We first evaluate the ID data $D_{\mathrm{ID}}$ to determine the OOD score threshold $\gamma$ that correctly identifies 95% of the ID samples, i.e.,

$$P(s({\mathbf{x}}_{\mathrm{ID}};\mathbf{f})\ge \gamma )=1.$$

(9)

We subsequently evaluate the auxiliary OOD data to identify samples with OOD scores lower than the threshold $\gamma$. These samples are selected as the auxiliary data for the training process, which can be formally formulated as follows:

$$D_{\mathrm{AUX}}^{\mathrm{train}}=\{{\mathbf{x}}_{\mathrm{AUX}}^i\in D_{\mathrm{AUX}}|s({\mathbf{x}}_{\mathrm{AUX}}^i;\mathbf{f})\le \gamma \}.$$

(10)

In practical implementations, it is often not necessary to employ the entire dataset $D_{\mathrm{AUX}}^{\mathrm{train}}$. Specifically, we can sort the data within $D_{\mathrm{AUX}}^{\mathrm{train}}$ based on their corresponding OOD scores in ascending order and then select a subset from the top, such as the top 10% of the data. This approach can significantly reduce computational costs. The experimental results presented in the following chapter demonstrate that even with the use of only the top 10% of the data, it is possible to achieve commendable performance levels.

Algorithm 1. FIS: Focus on Important Samples.

Require: The ID training dataset $D_{\mathrm{ID}}$; the auxiliary OOD dataset $D_{\mathrm{AUX}}$; the pre-trained classifier $\mathbf{f}$; the trade-off parameters $\alpha$. Ensure: The OOD detector $\mathbf{f}$.   1: # Setecting Auxiliary OOD Data:   2:   Compute OOD scores of the ID data $D_{\mathrm{ID}}$ to get the threshlod $\gamma$ via Equation (9);   3:    Compute OOD scores of the auxiliary OOD data $D_{\mathrm{AUX}}$;   4:    Select the auxiliary training OOD data $D_{\mathrm{AUX}}^{\mathrm{train}}$ via Equation (10);   5: # Training:   6:    for $e\leftarrow 1$ to $epochs$ do   7:       Fetch a mini-batch $({\mathcal{B}}_{\mathrm{ID}},{\mathcal{B}}_{\mathrm{AUX}})$ from $(D_{\mathrm{ID}},D_{\mathrm{AUX}}^{\mathrm{train}})$;   8:       if selecting then   9:          Perform steps 2 to 4.; 10:     end if 11:     Update $\mathit{\theta }$ by minimizing $\widehat{R}(\mathbf{f};\mathit{\theta })$ via Equation (6); 12:   end for 13: return The well-trained $\mathbf{f}(·;\mathit{\theta })$.

During training, we follow the same approach as OE, minimizing the empirical risk $\widehat{R}(\mathbf{f};\mathit{\theta })$, where we employ the multi-class cross-entropy loss as ${\ell }_{\mathrm{ID}}$, and adopt the loss function used in the classic OE method as ${\ell }_{\mathrm{OOD}}$, i.e., the cross-entropy from the model output distribution $\mathbf{f}\left(\mathbf{x}\right)$ to the uniform distribution.

5. Results

In this section, we evaluate the performance of our proposed method, FIS, across various OOD detection scenarios. The experimental outcomes consistently demonstrate the superior efficacy of FIS.

5.1. Experiment Setup

We begin by detailing our experimental setup.

Datasets:

• ID datasets: CIFAR-10 [46] and CIFAR-100 [46].
• OOD datasets: SVHN [47], Places365 [48], LSUN-Resize [49], Tiny-ImageNet-200 [50], TinyImage-300k [36], LSUN-Crop [49], and Textures [51].

Metrics:

• FPR95: False positive rate at 95% true positive rate.
• AUROC: The area under the receiver operating characteristic curve.
• ID ACC.: The classification accuracy on ID data.

Baselines:

• Post-hoc methods: GODIN [52], MSP [10], ReAct [17], Energy [16], Mahalanobis [15], MaxLogit [35], ODIN [13], KNN [18], DICE [53], Gram [30], GradNorm [21], ViM [31], and ASH [54].
• Fine-tuning methods: OE [36], CSI [55], SSD+ [56], MixOE [57], LogitNorm [58], MOS [29], ATOM [39], and VOS [6].

Backbones: We employ the Wide-ResNet-40-2 [59] as the backbone network for the main experiments. We conduct an ablation study by employing ResNet-18 [60].

Training Details: We train the model using stochastic gradient descent [61] and set the epoch to be 10, the initial learning rate to be 0.001 with cosine decay [62], and the batch size to be 128 for both ID and auxiliary training OOD data. We utilize the MLP score as the OOD score to identify important OOD data and to evaluate OOD detection performance. The selection strategy can be adjusted based on the type of auxiliary OOD data employed. By default, the selection strategy involves using OOD scoring to identify auxiliary OOD data prior to training. We will further elaborate on this in the subsequent discussion.

5.2. Main Results

CIFAR Benckmarks: The results on CIFAR benchmarks with auxiliary OOD data Tiny-ImageNet-200 are presented in

Table 2. Comparison between FIS and advanced methods on the CIFAR-10 benchmark. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred. Results are percentages. Bold font indicates the best results in a column.

	Method	Average		ID ACC.
		FPR95 ↓	AUROC ↑
	MSP [10]	50.04	91.12	94.84
	Mahalanobis [15]	35.66	93.34	94.84
	ODIN [13]	36.40	90.61	94.84
	GODIN [52]	23.97	95.55	94.84
	Energy [16]	32.87	91.73	94.84
Post-hoc	ReAct [17]	36.77	91.48	94.84
	KNN [18]	30.77	94.15	94.84
	DICE [53]	34.23	90.16	94.84
	Gram [30]	74.41	70.62	94.84
	GradNorm [21]	35.01	90.86	94.84
	ViM [31]	34.61	90.15	94.84
	ASH [54]	17.03	96.15	94.84
	OE [36]	12.41	97.85	94.07
	CSI [55]	17.36	96.91	94.17
	SSD+ [56]	14.84	97.36	94.71
	LogitNorm [58]	14.23	94.93	94.30
Fine-tuning	MOS [29]	65.37	74.34	94.83
	ATOM [39]	11.87	97.62	94.51
	MixOE [57]	13.55	97.59	94.51
	VOS [6]	31.55	91.56	93.67
	FIS (Ours)	${\mathbf{10.02}}_{\pm 0.1}$	${\mathbf{97.93}}_{\pm 0.0}$	${94.73}_{\pm 0.1}$

and

Table 3. Comparison between FIS and advanced methods on the CIFAR-100 benchmark. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred. Results are percentages. Bold font indicates the best results in a column.

	Method	Average		ID ACC.
		FPR95 ↓	AUROC ↑
	MSP [10]	79.94	75.55	75.96
	Mahalanobis [15]	59.63	82.03	75.96
	ODIN [13]	75.86	76.38	75.96
	GODIN [52]	76.57	82.23	75.33
	Energy [16]	72.12	79.69	75.96
Post-hoc	Gram [30]	65.71	72.46	75.96
	GradNorm [21]	71.45	79.55	75.96
	ViM [31]	69.72	80.43	75.96
	ReAct [17]	68.44	82.01	75.96
	KNN [18]	76.47	77.08	75.96
	DICE [53]	70.65	79.39	75.96
	ASH [54]	42.81	87.77	75.96
	OE [36]	44.95	89.56	74.99
	CSI [55]	70.26	81.44	69.90
	SSD+ [56]	56.65	87.38	74.21
	LogitNorm [58]	55.61	81.59	75.34
Fine-tuning	MOS [29]	57.67	79.60	74.98
	ATOM [39]	42.79	89.81	75.62
	MixOE [57]	52.04	86.46	74.36
	VOS [6]	73.43	79.98	73.72
	FIS (Ours)	${\mathbf{40.94}}_{\pm 0.7}$	${\mathbf{90.16}}_{\pm 0.2}$	${75.84}_{\pm 0.2}$

, where FIS outperforms all other approaches, thereby demonstrating its reliability and superiority. We highlight two key points: firstly, FIS employs only a small amount of auxiliary OOD data, indicating that more training data do not necessarily lead to better performance. The use of a limited yet more helpful and relevant subset of data can achieve performance comparable to, or even surpass, that of using a larger dataset. Secondly, while enhancing the OOD detection capabilities of the model, FIS also maintains high classification accuracy on ID data. This is attributable to the fact that the auxiliary OOD data we use possess more distinguishable features compared to ID data, thus not interfering with the training of ID tasks.

Near-OOD Setups: To assess the performance of our method under more challenging near-OOD detection scenarios, we conduct experiments using CIFAR-10 as the ID data, Tiny-ImageNet-200 as the auxiliary OOD data, and CIFAR-100 as the test OOD data. The experimental results are detailed in

Table 4. Comparison between FIS and advanced methods on the challenging near-OOD setting. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred. Bold font indicates the best results in a column.

	Method	FPR95 ↓	AUROC ↑
	MSP [10]	50.92	87.91
	Mahalanobis [15]	55.20	83.61
	ODIN [13]	59.08	85.35
	Energy [16]	56.26	87.15
	ReAct [17]	53.14	87.82
Post-hoc	Gram [30]	87.51	57.62
	GradNorm [21]	81.30	74.73
	ViM [31]	54.23	87.69
	MaxLogit [35]	56.16	87.19
	DICE [53]	66.91	83.19
	ASH [54]	88.40	74.75
	OE [36]	53.55	90.40
	CSI [55]	45.64	87.64
	SSD+ [56]	46.87	90.50
Fine-tuning	LogitNorm [58]	71.65	76.81
	MOS [29]	59.60	79.62
	ATOM [39]	51.87	90.73
	FIS (Ours)	40.70	91.48

. Our method once again demonstrates superiority over alternative approaches, thereby validating its robustness and reliability under extreme conditions. This is due to the same rationale: the auxiliary OOD data selected by our method exhibit more distinguishable features compared to ID data, which enables the model to more effectively learn to differentiate between ID and OOD data.

Backbones: To evaluate the performance of our approach on different models, we conducted an ablation study using ResNet-18 as the backbone network with CIFAR-10 as the ID data and Textures as the test OOD data. The results of these experiments are presented in

Table 5. Comparison between FIS and advanced methods on the CIFAR-10 benchmark when ResNet-18 serves as the backbone network. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred. Bold font indicates the best results in a column.

	Method	Average		ID ACC.
		FPR95 ↓	AUROC ↑
	OE [36]	14.14	97.57	94.01
	CSI [55]	31.47	89.43	90.25
	LogitNorm [58]	24.53	93.65	94.01
ResNet-18	MOS [29]	77.26	71.46	94.67
	ATOM [39]	13.76	96.87	94.07
	MixOE [57]	34.79	90.77	94.20
	FIS (Ours)	12.73	97.83	94.05

. The findings indicate that our method maintains an advantage when ResNet-18 serves as the model architecture, thereby substantiating the robustness of our approach across different model configurations.

Scores: The main idea of our method is to select important data from an auxiliary OOD dataset that are more conducive to OOD detection training. Although our previous experiments were conducted based on the MLP score, our new approach is not limited to this particular OOD score. To further explore our idea, we conducted experiments using different OOD scores on two CIFAR benchmarks. The results are depicted in

Table 6. The experimental results across OOD scores. The results are expressed as percentages. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred.

Scoring	Method	CIFAR-10			CIFAR-100
		FPR95 ↓	AUROC ↑	ID ACC. ↑	FPR95 ↓	AUROC ↑	ID ACC. ↑
MSP	OE	25.55	95.79	94.93	76.41	74.76	75.60
	FIS	21.41	96.35	94.24	68.44	78.42	74.94
MLP	OE	14.14	97.18	94.81	67.73	81.28	75.51
	FIS	13.91	97.24	94.59	65.23	84.43	74.10
Energy	OE	13.75	97.26	94.83	66.80	81.42	74.53
	FIS	14.22	97.09	94.88	60.31	86.39	73.85

, where Places365 serves as the test OOD data. Across both ID datasets, employing the MLP, MSP, and energy scores to filter more critical OOD detection data yielded better performance than using the entire dataset. It is noteworthy that our method is agnostic to the scoring function; different scoring functions produced consistent outcomes.

Additionally, the scoring function used for data selection can differ from the one used during the testing phase, as demonstrated by the experimental results shown in

Table 7. The results using different scoring for selecting auxiliary OOD data and for testing. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred.

Scoring for Selection	Scoring for Testing	FPR95 ↓	AUROC ↑	ID ACC. ↑
- ($D_{\mathrm{all}}$)	MLP	14.84	95.91	94.81
MSP	MLP	9.92	98.00	94.25
MLP	MLP	13.28	96.52	94.73

. The MSP+MLP results, which used the MSP score for data selection and the MLP score for testing, not only outperformed the outcomes from using all OOD data but also surpassed those from using the MLP score for both data selection and testing. This indicates that the data selection and training processes are distinct from each other; data selection is solely intended to identify data that are more beneficial for model training.

Ablation Study on Parameter $\gamma$: We conducted the ablation study on the parameter $\gamma$. The experimental results are summarized in Figure 3a, with CIFAR-10 as the ID data, TinyImage-300k as the auxiliary OOD data, and Places365 as the test OOD data. As observed, the training time cost decreased with an increase in $\gamma$, while the OOD detection performance also varied accordingly. Our default setting of $\gamma =0.95$ was chosen to minimize the training cost. However, if enhanced OOD detection performance is desired, the value of $\gamma$ can be further adjusted, for example, to 0.7 or 0.8.

Significance Test: To further substantiate that the advantages of our method are not attributable to random factors, we conducted additional significance analyses. To draw statistically robust conclusions, multiple replicates of the experiments were essential. Therefore, we focused on comparing our method with two closest-performing methods, OE and ATOM. The experimental setup mirrored that used in the CIFAR-10 benchmark. The experimental results are presented in Figure 3b, where each method was subjected to 30 independent replicates. We employed the paired-sample t-test to evaluate the statistical significance of the results. The results demonstrate that our method significantly outperforms the others, thereby validating its effectiveness.

Cost Evaluation: We conducted further experiments to compare the time–cost required by our method and other methods. The results are presented in

Table 8. Comparison of time–cost between FIS and advanced methods.

Method	CIFAR-10				CIFAR-100
	OE	MixOE	ATOM	FIS (Ours)	OE	MixOE	ATOM	FIS (Ours)
Time Cost (s)	191.42	230.42	117.65	58.30	191.97	251.76	104.73	48.45

. It can be observed that the time required for fine-tuning the model using the comparison methods was more than twice that of our method. This demonstrates that our method achieves excellent OOD detection performance while significantly reducing the training cost.

Selection Strategy: We also conducted further ablation studies on our selection strategy. The results are presented in

Table 9. Ablation study on the selection strategy. $\uparrow (\mathrm{or}\downarrow )$ indicates larger or smaller values are preferred.

Select	Tiny-ImageNet-200			TinyImage-300k
	FPR95 ↓	AUROC ↑	ID ACC. ↑	FPR95 ↓	AUROC ↑	ID ACC. ↑
Without selection	19.27	95.12	94.78	15.32	96.21	94.29
First select	18.75	96.46	94.04	13.12	97.28	94.58
Second select in epoch 5	21.17	96.08	94.56	11.02	97.49	94.97

, where CIFAR-10 serves as the ID dataset. Tiny-ImageNet-200 contained 70,000 training samples, while TinyImage-300k was a more complex dataset with 300,000 training samples. For the relatively simpler auxiliary OOD dataset, more frequent selections can lead to increasingly refined choices, which may cause overfitting of the model. In contrast, for more challenging and complex datasets, more frequent data selection during training is beneficial. Therefore, the strategy for selecting auxiliary OOD data is dependent on the characteristics of existing datasets.

5.3. Discussion

Here, we summarize the strengths and limitations of our method. Strengths: (a) Our method can train the model using a smaller subset of data that is more beneficial for training, thereby reducing the training cost. (b) Since the selected data are more advantageous for OOD detection fine-tuning, our method achieves better performance compared to other approaches. (c) Our method allows the use of different OOD scoring functions for data selection rather than being restricted to a single OOD scoring function, thus providing flexibility in adjustment. Limitations: (a) Although our method uses a smaller and more beneficial dataset for training, similar to other methods that employ auxiliary OOD data, it still requires the initial collection of a pool of auxiliary OOD data for selection purposes. (b) For simpler datasets, overly frequent data selection may pose a risk of overfitting.

6. Conclusions

In this paper, we conducted experimental analyses to identify data that are more beneficial for OOD detection training. We found that OOD data correctly detected by a well-trained classifier are more advantageous. This is because they possess features that are more distinguishable from ID data. As a result, they can help the model learn to differentiate between ID and OOD data without negatively impacting the classification learning of ID data. Furthermore, fine-tuning the model using only a small subset of the most beneficial data reduces the training cost. Based on these findings, we proposed a method called FIS, which uses OOD scores to select training data from OOD datasets. Our extensive experiments demonstrated the reliability and superiority of FIS. We believe our work will provide valuable insights for future research in the field of OOD detection.