Conformal Prediction for Counterfactual Detection in Concept Learning from Synthetic Visual Patterns

Abstract

Reliable detection of previously unseen classes under distributional shift remains a central challenge in concept learning and explainable artificial intelligence. In particular, high-performance deep learning models often lack statistically grounded mechanisms to signal when an instance deviates from learned concepts. This paper addresses this limitation by investigating whether conformal prediction can be effectively combined with a YOLOv5 deep learning classifier to enable principled counterfactual detection without prior exposure to the counterfactual class. As a controlled testbed, we employ Kandinsky patterns, a structured benchmark widely used in explainable AI research due to its rule-based generative transparency and suitability for concept learning studies. The proposed framework first classifies valid and invalid patterns and subsequently applies inductive conformal prediction to obtain calibrated prediction sets at a user-defined significance level. Counterfactual instances are, at start, identified based solely on information from known true and false patterns, without explicit training examples of the counterfactual class. Experimental results demonstrate that the conformalized detector reliably identifies a substantial proportion of previously unseen counterfactual patterns while maintaining statistical validity. In addition, the method flags unlabeled (“empty”) instances, thereby providing a principled signal for the emergence of new concepts. By conformalizing YOLOv5 outputs, the approach establishes a statistically sound mechanism for uncertainty-aware detection of divergent classes, contributing to robust and explainable concept learning in structured visual pattern recognition.

1. Introduction

Machine Learning has matured from isolated algorithmic techniques for pattern recognition into a unifying scientific foundation that underpins many of today’s most influential advances in Artificial Intelligence [1]. Ensuring the reliability, robustness, and interpretability of predictive models remains a fundamental challenge, marked by class ambiguity, imprecision and uncertainty [2].

Conformal prediction (CP) has emerged as a statistically principled framework offering formal guarantees of prediction reliability addressing these challenges. Unlike traditional point prediction methods that often lack transparency or probabilistic interpretability, CP constructs prediction sets with user-specified confidence levels while requiring only the weak assumption of data exchangeability [3,4,5,6].

This continual evolution of CP has brought forward advances that have expanded its applicability and effectiveness. Innovations such as Conformal Predictive Distributions [7], conformal change-point detection [8], Venn–Abers predictors [9], and enhancements like jackknife+ [10] have opened new frontiers in CP research [11]. These advancements not only improve the theoretical underpinnings of CP but also broaden its practical utility across a range of challenging tasks in Pattern Recognition.

The relevance and growing interest in CP are evident from its increasing presence in the academic literature [12].

This paper contributes to this expanding field by presenting an innovative application of CP in the analysis of Kandinsky patterns, a synthetic test dataset designed for concept learning and explainability in artificial intelligence [13]. By integrating CP with the Yolo v5 network model [14,15], this study demonstrates how CP can enhance the identification of true, false, and counterfactual patterns within the dataset. The results not only highlight the efficacy of CP in detecting previously unseen data but also underscore its potential as a robust framework for identifying novel classes in complex datasets. This research aligns with the broader goal of advancing Pattern Recognition methodologies through the integration of explainable and statistically rigorous approaches like conformal prediction.

The versatility and statistical rigor of CP naturally align with the broader objectives of Explainable AI (XAI), where transparency and interpretability are paramount. In the pursuit of trustworthy AI, understanding and communicating why models make certain predictions is as crucial as the predictions themselves. Counterfactual reasoning plays a pivotal role in this domain, enabling insights into what-if scenarios that reveal how specific features or inputs influence outcomes [16]. The concept of counterfactual reasoning is particularly valuable in identifying anomalies, novel patterns, and previously unseen classes within data. Counterfactuals are also very important for another reason: by asking “what if?” people can familiarize themselves with unfamiliar processes by understanding the hypothetical input conditions under which the output changes. This not only promotes explainability but also confidence and trust in the results [17].

Kandinsky patterns are designed as synthetic datasets for benchmarking concept learning and explainability, providing an ideal testbed for exploring these challenges. Their structured yet abstract nature offers a unique framework for studying counterfactual reasoning in AI, making them an excellent application domain for the advanced capabilities of CP.

This paper introduces a novel methodology for the analysis of Kandinsky patterns by integrating the Yolo v5 network model with conformal prediction. The core aim of the study is to enhance the accuracy in distinguishing between true and false Kandinsky patterns, with a particular focus on identifying counterfactual Kandinsky-like images. The innovation of this research lies in the application of conformal prediction to detect these counterfactual patterns using, initially, only the information derived from true and false patterns, without prior knowledge of the counterfactuals themselves.

The findings of the study demonstrate that a substantial proportion of counterfactual Kandinsky patterns can be successfully identified using the proposed approach. This underscores the efficacy of the method in flagging patterns that fall outside the model’s established applicability domain. A key advantage of the approach is the ability of the conformal prediction framework to operate at a user-defined significance level, enabling it to detect non-labeled (empty) images that are distinct from both true and false Kandinsky classes. These non-labeled images are indicative of the emergence of a new, previously unseen class.

The paper concludes that applying conformal prediction to the outputs of the Yolo v5 network allows for the statistically rigorous identification of this new counterfactual class. This finding highlights the potential of conformal prediction in recognizing novel and divergent classes within data, contributing to the broader field of pattern recognition and anomaly detection. The approach presented here not only advances the understanding of Kandinsky patterns but also offers a robust framework for the detection of new and unseen classes in various domains.

Recent developments in YOLO-based detection architectures demonstrate the continued evolution of real-time visual inspection systems. For example, Wang et al. [18] propose a fusion of enhanced Canny–Devernay edge detection with YOLOv6 for real-time defect detection in metal components, showing how modern YOLO variants can be integrated with classical image processing techniques in structured industrial inspection tasks. Such hybrid detection pipelines underline the adaptability of contemporary YOLO frameworks for structured pattern recognition. In contrast to performance-driven detection optimization, our work extends this line of research by incorporating conformal prediction to obtain statistically valid uncertainty quantification and principled counterfactual detection under distributional shift.

2. Background and Related Work

2.1. Conformal Prediction

Conformal prediction is a machine learning method that enables the development of valid predictions under weak probabilistic assumptions such as exchangeability, i.e., invariant under permutations, and estimates the confidence of predicting the class label, y, for a new example, x, given the training set examples:

$$z_1=(x_1,y_1),z_2=(x_2,y_2),\dots ,z_l=(x_l,y_l).$$

In the inductive learning setting of CP used in this investigation, this entails splitting the training set into a proper training set $(z_1,\dots ,z_n)$ and a calibration set and finding out how strange (non-conformal) a new example is in comparison to the calibration set examples $(z_{n+1},\dots ,z_l)$ by calculating the non-conformity measure (NCM) ${\alpha }_i$ of $z_i$ for the available classes (j). The corresponding p-value is then calculated according to Equation (1):

$$p_y^j=|\{i=n+1,\dots l+1:y_i=y,{\alpha }_i>{\alpha }_{l+1}\}|/|\{i=n+1,\dots ,l+1:y_i=y\}|$$

(1)

The class is included in the predicted label set if j: $p_y^i>\epsilon$, where $\epsilon$ is the set significance level. In the Mondrian setting used in this investigation, a separate p-value is calculated for each class using the respective class-specific calibration set. For a more detailed description, see reference [19].

Image classification produces a classifier that outputs scores for a set of possible classes. In many cases, such classifiers do not output a measure of confidence or uncertainty along with their predictions. Consequently, conformal prediction provides a guaranteed measure of confidence: specifically, given a user-defined significance level $ϵ$, the conformal predictor generates a prediction set that contains, at most, $ϵ$% errors [20]. Conformal prediction uses a set of calibration data to derive a conformal p-value that ensures this error level. For more details on the calibration procedure, see Alvarsson et al. (2021) [21]. The guaranteed error rate is achieved by the conformal prediction framework by providing a set of predictions from the classifier, not necessarily a single prediction value. For the Kandinsky patterns challenges, there are three categories (True, False, and Counterfactual). The three-category CP classification problem in this investigation can have eight possible prediction outcomes: {T}, {F}, {Cf}, {T,F}, {T,Cf}, {F,Cf}, {T,F,Cf} and {empty}. The first three predictions are single-label outcomes, the following four predictions are multi-label outcomes, and the remaining prediction is “empty” (no category assigned).

Two key measures in conformal prediction are validity and efficiency.

Validity is the percentage of correct predictions in conformal prediction at the set significance level where the prediction contains the correct category. The {T,F,Cf} classification prediction is thus always correct, as it contains all possible available categories. However, it provides very little useful information to the user. Therefore, efficiency is also an important measure for the system.

Efficiency, for each category, is defined as the percentage of single-category predictions (only one category), regardless of whether the prediction is correct or not, at the set significance level. High efficiency is therefore desirable since a larger number of single-category predictions are more informative as they contain only one category [22].

The CP framework can also return an “empty” classification. This classification is always erroneous, as it contains no categories, and therefore cannot contain the correct prediction. However, it provides more useful information than a system that, for such cases, predicts a classification with a very low confidence level. This article demonstrates the potential for using the non-single classes as an indicator that a new type of data has been introduced to the classifier.

In this investigation, the Yolo v5 network was used as the underlying classifier, but the use of CP using neural networks has previously been investigated by Papadopoulos et al. (2007) [23].

Karimi et al. (2024) [24] conducted a comprehensive evaluation of deep learning classifiers within a conformal prediction (CP) framework, focusing on architectures such as ResNet, DenseNet161, and VGG16 applied to the ImageNet dataset comprising 1000 classes. Their work highlights the performance characteristics of these models in large-scale image classification tasks under CP constraints. In a related study, Katsios et al. (2024) [25] explored multi-label classification within the CP framework, employing the Mahalanobis distance as a nonconformity measure. Their experiments, conducted on benchmark datasets such as Emotions and Yeast, provided insights into the applicability of distance-based measures for multi-label uncertainty quantification. Complementing these efforts, Carlevaro et al. (2024) [26] proposed a probabilistic scaling approach to CP, applying it across several binary classification tasks using datasets such as MNIST [27], CIFAR-10, Pneumonia MNIST, Fashion MNIST [28] or WikiArt. Their findings contributed to understanding the generalizability and calibration properties of CP across diverse image domains [26].

2.2. Kandinsky Patterns

Kandinsky patterns are synthetic datasets designed for evaluating and benchmarking AI systems, particularly in the domains of concept learning and explainable AI (XAI). There remains a significant disparity between the ability of AI to recognize patterns and its capacity to learn concepts at a level comparable to that of humans, and there is considerable interest in the use of experimental settings and diagnostic/benchmark datasets to scrutinize current methodologies and propel advancements in pattern analysis and machine intelligence towards concept learning [29,30,31]. Kandinsky patterns are one of a number of approaches available for benchmarking concept learning, reasoning, and generalization. Reasoning and explanation have a long history within the AI/machine learning community [32,33] and recently quite a number of authors proposed mechanisms for generating explanations by deep learning models, and there is ongoing advancement in diagnostic and benchmark datasets, including Compositional Language and Elementary Visual Reasoning diagnostics dataset (CLEVR) [34], CLEVERER [35], CLOSURE [36], CURI [37], Bongard-LOGO [38], and V-PROM [39].

Kandinsky patterns are considered interpretable in the sense that they are generated from explicitly defined symbolic rules governing color, shape, and spatial relations. Each image instance can be traced back to a formally specified logical condition that determines its validity with respect to a given concept. Unlike natural image datasets, where semantic structure is often implicit and high-dimensional, the generative process of Kandinsky patterns is transparent and rule-based, enabling a direct mapping between visual configuration and symbolic concept description. This interpretability claim therefore refers to the structural transparency of the dataset rather than to a formally conducted human-subject study. The present work focuses on methodological evaluation in a controlled setting; empirical investigation of human-perceived interpretability and cognitive difficulty constitutes an important direction for future research.

Kandinsky patterns consist of geometrically structured figures characterized by attributes such as shape, size, color, and spatial arrangement [13]. They are defined by underlying logical rules or relationships that define a set of “true” patterns while distinguishing them from “false” or “counterfactual” ones. For example, a Kandinsky pattern may define a rule such as “two shapes of the same type but different colors must appear in the image.” AI models tasked with learning such rules are evaluated not only on their ability to classify patterns correctly but also on their capacity to generalize these rules to unseen instances. This capability mirrors human conceptual reasoning, where abstract relationships are inferred from limited examples. Their structured yet abstract nature makes them ideal for exploring the capacity of AI systems to generalize, reason, and learn abstract concepts, beyond simple pattern recognition. What is special about the Kandinsky patterns is that they can be meaningfully recognized by both humans and machines, which makes them suitable as intelligence tests for both humans and machines [40].

Conformal prediction offers a robust framework for analyzing Kandinsky patterns, particularly in the detection of counterfactuals and novel classes. By providing prediction regions with valid statistical guarantees, CP enables the rigorous identification of patterns that fall outside the domain of previously observed data. For Kandinsky patterns, this means CP can flag instances that do not conform to the learned rules as “nonconforming” or “counterfactual,” effectively identifying potential new classes or anomalies.

The integration of CP with Kandinsky patterns bridges concept learning and explainability by ensuring both accuracy and interpretability. CP’s statistical guarantees provide confidence in the model’s predictions, while its ability to highlight nonconforming instances supports the detection of novel concepts. This combination underscores the potential of CP as a tool for advancing research in XAI and for fostering a deeper understanding of how AI systems learn, reason, and generalize abstract concepts. By applying conformal prediction to Kandinsky patterns, researchers can evaluate not only the accuracy of AI systems but also their ability to operate transparently and reliably in uncertain and novel scenarios.

2.3. Definition of Kandinsky Patterns

A Kandinsky Figure is a square image containing one to n geometric objects. Each object is characterized by its shape, color, size and position within this square. Objects do not overlap and are not cropped at the border. All objects must be easily recognizable and clearly distinguishable by a human observer.

The set of all possible Kandinsky Figures k is defined by the general definition together with a specific set of values for shape, color, size, position and the number of geometric objects. In the following examples we use for shape the values circle, square and triangle; for color we use the values red, blue, yellow, and we allow arbitrary positions and size with the restriction that it is still recognizable. Furthermore, we require each Kandinsky Figure to contain exactly four objects in the following illustrative example (see Figure 1).

Figure 1. A Kandinsky Figure with 4 distinct shapes: A large blue circle positioned slightly left of center. A large yellow circle located to the lower right of the blue circle. A red triangle in the lower left area, oriented with its base at the bottom and apex pointing upward. A small red square in the upper right quadrant.

In the demo implementation this fact is embedded in the base class “Kandinsky Universe”, and in the generator functions https://github.com/human-centered-ai-lab/app-kandinsky-pattern-generator (accessed on 13 March 2026).

All generated images are openly available in full resolution in our public GitHub repository for full reducibility of the experimental setup. Since the primary contribution of this work is methodological—namely, the integration of conformal prediction with a deep learning classifier for statistically valid counterfactual detection, the visual examples themselves are not essential for understanding the technical framework or the reported quantitative results. Readers interested in inspecting the complete set of pattern instances can access them directly via the repository.

A Statement $s\left(k\right)$ about a Kandinsky Figure k is either a mathematical function, $s\left(k\right)\to B$, with $B(0,1)$, or a natural language statement $n\left(k\right)$, which is either true or false.

A Kandinsky pattern K (Figure 2) is defined as the subset of all possible Kandinsky Figures k with $s\left(k\right)\to 1$ or the natural language statement $n\left(k\right)$ is true. $s\left(k\right)$ and a natural language statement $n\left(k\right)$ are equivalent if and only if the resulting Kandinsky patterns contains the same Kandinsky Figures. Both $s\left(k\right)$ and the natural language statement $n\left(k\right)$ define the ground truth of a Kandinsky pattern.

Figure 2. Five Kandinsky Figures form a Kandinsky pattern, see explanation examples in Figure 1.

In a deep learning solution, a classification algorithm for a visual pattern is usually represented as a highly non-linear, high-dimensional network. One aim of explainable AI is to identify areas of activation within the network structure, which correspond to concepts in the natural language statement.

Problem 1: How can we explain a Kandinsky pattern, if we do not know the ground truth and the membership of Kandinsky Figures to a Kandinsky pattern is only known for a limited number of Kandinsky Figures?

Problem 2: Generate a natural language statement which is easily understandable and equivalent to the machine explanation (classification algorithm).

The process of explanation is the generation and refinement of a hypothesis to find the underlying description. The validation is achieved by the scientific method of asking a question, forming a testable hypothesis, setting up the experimental design, running the experiment and either accepting the hypothesis, rejecting it or, in the third case, according to [41], one cannot make any assumption.

The ground truth is used to prove or disprove research hypotheses. “Ground truthing” consequently refers to the process of collecting the proper objective (provable!) data for testing the hypothesis. For a machine learning algorithm, an explanation can be seen as the successful classification algorithm of a Kandinsky pattern.

The following example illustrates the above:

The ground truth $gt\left(k\right)=$ “the Kandinsky Figure has two pairs of objects with the same shape, in one pair the objects have the same color, in the other pair different colors, two pairs are always disjunct, i.e., they don’t share objects” defines the Kandinsky Pattern $K_{gt}$.

For a more general hypothesis $h_1\left(k\right)$ = “the Kandinsky Figure has two pairs of objects with the same shape”, we see that $K_{h1}\setminus K_{gt}\ne \varnothing$, i.e., the Kandinsky pattern of $h_1\left(k\right)$ contains Kandinsky Figures which are not in the Kandinsky pattern of the ground truth. Figure 3 shows Kandinsky Figures according to $h_1\left(k\right)$; the first row is a contradiction to the ground truth, i.e., it falsifies $h_1\left(k\right)$.

Figure 3. Kandinsky Figures of $h_1\left(k\right)$; the first row shows counterfactuals. For examples of explanations of the colors and shape refer to Figure 1.

A specific hypothesis like $h_2\left(k\right)$ = “the Kandinsky Figure consists of two triangles with different color and two circles of same color” generates a Kandinsky pattern $K_{h2}$ with $K_{gt}\setminus K_{h2}\ne \varnothing$, i.e., the Kandinsky pattern of $h_2\left(k\right)$ is missing Kandinsky Figures which are in the Kandinsky pattern of the ground truth. Figure 4 shows in the first row Kandinsky Figures according to $h_2\left(k\right)$ and Kandinsky Figures from $K_{gt}$ in the second row, which falsify $h_2\left(k\right)$. $K_h\setminus K_{gt}\cup K_{gt}\setminus K_h$ is the set of counterfactual Kandinsky Figures for a given hypothesis h.

Figure 4. Kandinsky Figures of $h_2\left(k\right)$ in the first row and Kandinsky Figures from $K_{gt}$ in the second row, which falsify $h_2\left(k\right)$. For example explanations of shape and colors refer to Figure 1.

In the context of this work we evaluated three open-source Kandinsky datasets, defined in [13].

2.4. Dataset 1—Objects and Shapes

The first dataset, referred to as Objects and Shapes, was generated from a formally specified rule defining the validity of a Kandinsky Figure. The ground truth was based on structured relations between large shapes and smaller objects placed within them. Small objects were arranged inside larger geometric shapes, and the type of large shape determined admissible object types and color combinations. In particular, no small object may share the same shape type as its enclosing large shape. Furthermore, each large shape restricted the allowed object colors according to predefined constraints. These rules together defined the concept that determined whether a configuration was valid or invalid.

Invalid patterns were constructed by violating one or more of these defining constraints while preserving visual plausibility. In addition, counterfactual instances were generated using alternative but systematically structured hypotheses that differed from the original ground-truth rule. These counterfactual configurations remained coherent and structured, yet did not satisfy the true generative concept.

All generated pattern instances, including valid, invalid, and counterfactual examples, are openly available in full resolution in our public GitHub repository. The repository contains the complete dataset, generation scripts, and documentation to ensure transparency and reproducibility. As the present work focuses on the methodological integration of conformal prediction with deep learning for statistically valid counterfactual detection, visual examples are not reproduced in the manuscript. Readers interested in inspecting the pattern configurations in detail are encouraged to consult the repository directly.

Download the dataset for challenge 1 here:  https://tinyurl.com/Kandinsky-C1 (accessed on 13 March 2026), https://github.com/human-centered-ai-lab/dat-kandinsky-patterns/tree/master/challenge-nr-1 (accessed on 13 March 2026).

2.5. Dataset 2—Nine Circles

The second dataset, referred to as Nine Circles, consisted of Kandinsky Figures composed of nine circles arranged in a regular grid structure. Each configuration followed a formally defined ground-truth rule that determined whether the pattern was valid or invalid. The rule specified structural and attribute-based constraints over the grid arrangement, such as color distributions and relational properties between neighboring circles.

Invalid instances were generated by violating one or more of these defining constraints while preserving the overall grid structure. In addition, counterfactual configurations were constructed to be “almost valid”: they satisfied a hypothesis that closely resembled the true rule but deviated in a systematic and conceptually meaningful way. These counterfactuals were therefore visually coherent and structurally plausible, yet they did not conform to the actual ground-truth definition.

The purpose of this dataset was to provide a controlled setting for studying concept discrimination, rule learning, and the detection of subtle deviations from established patterns. All generated configurations, including valid, invalid, and counterfactual instances, are openly available in full resolution in our public GitHub repository. The repository contains the complete dataset and the generation scripts to ensure transparency and reproducibility. As the emphasis of the present work lies on the methodological aspects of uncertainty-aware counterfactual detection, visual examples are not reproduced in the manuscript, and readers are encouraged to consult the repository for detailed inspection of the pattern instances.

2.6. Dataset 3—Blue and Yellow Circles

The third dataset, referred to as Blue and Yellow Circles, is composed of Kandinsky Figures containing circles of identical size in two colors: blue and yellow. The validity of each configuration was determined by a formally specified ground-truth rule governing the spatial arrangement and relational properties of the circles. That rule defined the concept to be learned and provided a precise criterion for distinguishing valid from invalid patterns.

Invalid instances were generated by systematically violating one or more elements of the defining rule while preserving the overall structural simplicity of the configuration. In addition, counterfactual examples were constructed to be structurally coherent and visually plausible but to follow an alternative hypothesis that differed from the true generative rule. These counterfactual configurations therefore appeared “almost valid” yet remained conceptually distinct from the ground truth.

This dataset provided a controlled environment for analyzing concept discrimination in settings with reduced visual complexity but non-trivial relational structure. All generated configurations—including valid, invalid, and counterfactual instances—are openly available in full resolution in our public GitHub repository. The repository contains the complete dataset and generation scripts to ensure full transparency and reproducibility. As the present study concentrates on the methodological integration of conformal prediction with deep learning for statistically valid counterfactual detection, the visual examples themselves are not reproduced in the manuscript. Readers interested in detailed inspection of the pattern instances are encouraged to consult the repository directly.

3. Methodology

3.1. Dataset Preparation

The datasets for the experiments described here were downloaded from:

https://github.com/human-centered-ai-lab/dat-kandinsky-patterns (accessed on 8 July 2025).

The datasets, excluding the counterfactual class, were randomly divided into training (70%), validation (10%) and test (20%) sets, respectively.

3.1.1. Kandinsky Challenge 1

The images not belonging to the Kandinsky patterns were down-scaled from 600 × 600 to 450 × 450 and centered. Background color was then added as a frame to restore the original size of the images. The images belonging to the Kandinsky patterns as well as the counterfactual images were used without modification.

3.1.2. Kandinsky Challenges 2 and 3

The images were used without modification.

3.2. Classification Model

The classification models were developed using the YOLO v5 network model [15,42] and code obtained from Ultralytics GitHub repository YOLO v5 on  https://github.com/ultralytics/yolov5 (accessed on 27 October 2022). The YOLO v5 architecture was intentionally used to safe computational costs. Comparison experiments with YOLO v10 showed not dramatically better results, but enormous longer computational costs.

The YOLO v5 architecture consists of four parts: input, backbone, neck, and output. The architecture is a modified CSPDarknet53 comprised of a strided convolution layer followed by image features extracting convolutional layers. The neck uses a spatial pyramid pooling fast layer (SPPF) and a modified Cross Stage Partial-Path aggregation network (CSP-PAN) for instance segmentation. Finally, the output (head) utilizes fine-tuned anchor boxes and convolutional layers that predict bounding boxes, class probabilities, and objectness scores. Output results in this investigation were presented as the scores from the network. The pretrained Yolo yolov5s-cls network and default settings (except #epochs = 100) were used for the fine tuning (training) of the models. The scoring functions from the YOLOV5 predictions for the calibration and test set, respectively, were used as non-conformity measures (NCM) and used for calculating the p-values of each class in the test set.

3.3. Analysis

The three challenges were all analyzed in a similar manner with respect to two different scenarios where a new collection of images that needed examination in order to determine their classes was used. In the first scenario all test set and remaining, undetected, counterfactual set images were presented to the classifier in each round, and the classifier was updated in an iterative manner after each round in order to identify counterfactual images. In the second scenario a much smaller set of 50 undetected counterfactual images were presented to the classifier each round, but otherwise, the same conditions were used as for scenario 1.

The procedure for the first scenario consisted of the following steps (see Algorithm 1):

1.

The network was trained using the current training and validation sets.

2.

The resulting “Best” model was used to predict the test set and the current undetected counterfactual set.

3.

Conformal prediction was used to analyze 50% of the test set, randomly selected, and determine the predicted labels and minimize the total number of non-single, i.e., multi-class and empty, class predictions (best solution). This was done by testing several different significance levels (% acceptable errors): 0.01, 0.02, 0.03, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3.

4.

Conformal prediction was then used to analyze the remaining part of the test set and the current undetected counterfactual set, using the “best-solution” significance level, to determine the predicted classes.

5.

The predicted counterfactual, multi-class or empty class images were “examined” (with a human in the loop) and if the true class was counterfactual, these images were randomly moved to the counterfactual training set (2/3) and the counterfactual validation set (1/3), respectively.

6.

Steps 1–5 completed one round. We then returned to step 1 until no further counterfactual images were detected by the procedure.

The procedure for the second scenario operated in the same manner as the first procedure with the exception that only 50 images of the remaining undetected counterfactual set were randomly selected and presented to the classifier in each round for a maximum of 45 rounds.

Algorithm 1 Iterative counterfactual detection with conformal prediction

Require: Dataset $\mathcal{D}={\mathcal{D}}^{\mathrm{true}}\cup {\mathcal{D}}^{\mathrm{false}}$, initial counterfactual set ${\mathcal{C}}_0$, significance grid        $\mathcal{E}=\{0.01,0.02,0.03,0.05,0.1,0.15,0.2,0.25,0.3\}$   1: Randomly split $\mathcal{D}$ into ${\mathcal{D}}_{\mathrm{train}}$, ${\mathcal{D}}_{\mathrm{val}}$, ${\mathcal{D}}_{\mathrm{test}}$ with sizes $(700,100,200)$ per class   2: $t\leftarrow 0$   3: ${\mathcal{C}}_{\mathrm{undetected}}\leftarrow {\mathcal{C}}_0$   4: repeat   5:       Train classifier $f_t$ on ${\mathcal{D}}_{\mathrm{train}}$   6:       Compute nonconformity scores on ${\mathcal{D}}_{\mathrm{val}}$   7:       Randomly split ${\mathcal{D}}_{\mathrm{test}}$ into ${\mathcal{D}}_{\mathrm{test}}^{\left(1\right)}$ and ${\mathcal{D}}_{\mathrm{test}}^{\left(2\right)}$ with equal size   8:       if Scenario 1 then   9:         ${\mathcal{C}}_{\mathrm{pred}}\leftarrow {\mathcal{C}}_{\mathrm{undetected}}$ 10:     else 11:         ${\mathcal{C}}_{\mathrm{pred}}\leftarrow$ random $5\%$ subset of ${\mathcal{C}}_{\mathrm{undetected}}$ 12:     end if 13:     for all $\epsilon \in \mathcal{E}$ do 14:         Apply conformal prediction with level $\epsilon$ on ${\mathcal{D}}_{\mathrm{test}}^{\left(1\right)}$ 15:         Compute number of multi-class and empty prediction sets: $$M\left(\epsilon \right)$$ 16:     end for 17:     Select optimal level $${\epsilon }^{*}=arg\underset{\epsilon \in \mathcal{E}}{min}M\left(\epsilon \right)$$ 18:     Apply conformal prediction with ${\epsilon }^{*}$ on ${\mathcal{D}}_{\mathrm{test}}^{\left(2\right)}\cup {\mathcal{C}}_{\mathrm{pred}}$ 19:     Identify counterfactual candidates $${\mathcal{C}}_{\mathrm{new}}=\{x\in {\mathcal{C}}_{\mathrm{pred}}:{\Gamma }_{{\epsilon }^{*}}\left(x\right)\mathrm{is}\mathrm{counterfactual},\mathrm{multi}\text{-}\mathrm{class}\mathrm{or}\mathrm{empty}\}$$ 20:     Randomly split ${\mathcal{C}}_{\mathrm{new}}$ into ${\mathcal{C}}_{\mathrm{train}}^{\mathrm{add}}$ (2/3) and ${\mathcal{C}}_{\mathrm{val}}^{\mathrm{add}}$ (1/3) 21:     Update datasets: $${\mathcal{D}}_{\mathrm{train}}\leftarrow {\mathcal{D}}_{\mathrm{train}}\cup {\mathcal{C}}_{\mathrm{train}}^{\mathrm{add}}$$ $${\mathcal{D}}_{\mathrm{val}}\leftarrow {\mathcal{D}}_{\mathrm{val}}\cup {\mathcal{C}}_{\mathrm{val}}^{\mathrm{add}}$$ 22:     ${\mathcal{C}}_{\mathrm{undetected}}\leftarrow {\mathcal{C}}_{\mathrm{undetected}}\setminus {\mathcal{C}}_{\mathrm{new}}$ 23:     $t\leftarrow t+1$ 24: until  ${\mathcal{C}}_{\mathrm{new}}=⌀$

4. Results and Discussion

The inductive conformal prediction scheme was used in this empirical study because of its much lower computational burden compared to full transductive conformal prediction. For the former approach, only one model needs to be developed and two sets, the test and calibration sets, need to be predicted by the model for each round in order for the conformal prediction output (p-values) to be computed. For the latter approach, on the other hand, a new model needs to be developed for each test set example. The results for the three challenges are depicted in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 and in Supplementary Materials (the underlying results for Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10).

Figure 5. Kandinsky results—first scenario: (a) challenge 1, (b) challenge 2, (c) challenge 3. Black: CF images remaining; red: not CF class but predicted CF; blue: CF predicted CF; grey: CF predicted multi- or empty.

Figure 6. Kandinsky results—first scenario (focused on detected images): (a) challenge 1, (b) challenge 2, (c) challenge 3. Black: CF images remaining; red: not CF class but predicted CF; blue: CF predicted CF; gray: CF predicted multi- or empty.

Figure 7. Kandinsky results—second scenario: (a) challenge 1, (b) challenge 2, (c) challenge 3. Black: CF images remaining; red: not CF class but predicted CF; blue: CF predicted CF; gray: CF predicted multi- or empty.

Figure 8. Kandinsky results—second scenario (focused on detected images): (a) challenge 1, (b) challenge 2, (c) challenge 3. Black: CF images remaining; red: not CF class but predicted CF; blue: CF predicted CF; gray: CF predicted multi- or empty.

Figure 9. Kandinsky—first scenario counterfactual (CF) predictions over all rounds: (a) challenge 1, (b) challenge 2, (c) challenge 3.

Figure 10. Kandinsky—second scenario counterfactual (CF) predictions over all rounds: (a) challenge 1, (b) challenge 2, (c) challenge 3.

Figure 5 and Figure 6 show that challenges 1 and 2 in the first scenario have a similar behavior in terms of identifying the counterfactual (CF) images, where the remaining CF images rapidly decrease until no unidentified CF images exist after round 4. In addition, the predicted multi-class and empty-class CF images are more or less identified in a similar manner. In the first round, there are no images predicted as CF, as this class has not yet been presented to the classifier. However, a certain number of non-single-class predictions occur. In fact, all these images are predicted as the “empty” class, indicating that they are too different from previously seen images and potentially may constitute some class never seen before. Inspection of these images discloses the existence of a new CF class. Only one image for the non-CF class is predicted as “empty” in challenge 2 and none in challenge 1. The identified CF images are then randomly added to the training or validation sets of the CF class. In round 2, the classifier, now potentially able to also identify the CF class, correctly identifies a large portion of the images as counterfactual. Also, as in round 1, almost all of the non-single class predictions are “empty” predictions (only six multi-class predictions in challenge 2) and subsequently inspected, classified and added to the CF training or validation sets. These non-single class predicted images contain a small portion of non-CF images (14 and eight, for challenges 1 and 2, respectively). The number of remaining, non-identified, CF images in round 3 markedly decreases in both challenges 1 and 2 to 276 and 12, respectively. In round 3 the remaining CF images are identified. Furthermore, we note that the images falsely identified as CF are kept at a very low level throughout all of the rounds for challenges 1 and 2; challenge 3 is different from the previous two challenges in that CF images are not identified as rapidly as previously seen. Even after 13 rounds, the classifier is still not able to identify the remaining 17 CF images, which also remain undetected in round 14 and predicted as “true” images. The detection performance values of predicted non-single class CF images are also more erratic. The number of images erroneously predicted as CF is somewhat larger in challenge 3 compared to challenges 1 and 2.

The same behavior detected in the first scenario is also present in the second scenario, as depicted in Figure 7 and Figure 8, where a considerable smaller portion of the CF set (50 images) is predicted by the classifier in each round. After 22 rounds, all of the CF images have been identified for challenges 1 and 2, respectively, while only close to 61% of the CF images have been identified for challenge 3.

It should be noted that the minimum number of rounds possible for detecting 1000 CF images by presenting 50 such images to the classifier each round is 20. After 44 rounds, all but seven CF images have been identified in challenge 3. Also, a similar, more erratic behavior is seen in challenge 3 compared to challenges 1 and 2, respectively, in the second scenario.

The present iterative approach represents a pragmatic recalibration procedure to detect new, unseen, classes using the CP framework. Optimizing the significance level by minimizing the non-single predictions enables the successful identification of CF images. The significance level $ϵ$ is set at various levels during the loops for scenario 1 and 2, respectively. In general, for scenario 1, where all undetected CF images are presented to the classifier in every round, the “best” significance levels are 0.01 and 0.1 for challenge 1 and 0.01 and 0.03 for challenge 2, respectively. The somewhat different challenge 3 has the “best” significance levels spread across most of the tested levels with a slight tendency towards the higher levels from 0.1 and above. For scenario 2, where only 50 undetected CF images are presented to the classifier in every round, the “best” significance levels are 0.01, 0.02 and 0.03 for challenges 1 and 2. For challenge 3 the situation is the opposite, and the three highest levels (0.2, 0.25 and 0.3) dominate with 42 out of 44 selections. At the same time this pragmatic procedure does violate the validity in a number of cases, particularly for the CF class, depending on the number of CF images presented to the classifier in each round (scenario 1 or 2). From Figure 9 it is evident that most of the non-single predictions are “empty” predictions for challenges 1 and 2 while a majority represents multi-class predictions in challenge 3. The result is that the validity is violated for the CF class. However, for scenario 2, the results are different for challenges 1 and 2 in that many more CF images are predicted as CF (Figure 10). For challenges 1 and 2, all rounds, apart from round 1 where the classifier has not been presented the CF class and round 2, are valid or close to it. The more erratic challenge 3 still suffers from invalid CF predictions for the CF class because the number of detected CF images is well below the 50 images presented in each round (median: 48.0% and average: 47.7%). The corresponding numbers for challenges 1 and 2 are 100 and 94.6% and 100 and 95.2%, respectively. The average prediction sets for both scenarios and all three challenges are around one and in a majority of cases, below one (see Supplementary Material). This indicates that many predicted test set images are assigned to the “empty” class, which was the objective of the proposed procedure used in the investigation to detect new “strange” (different) images.

5. Conclusions and Future Outlook

Our study investigated the integration of conformal prediction (CP) with the YOLOv5 deep learning architecture for the analysis of Kandinsky patterns, with particular emphasis on distinguishing true, false, and counterfactual patterns. By coupling CP with a high-capacity object detection model, we demonstrated a principled mechanism for identifying samples that fall outside the model’s applicability domain. The results showed nicely that CP reliably flags counterfactual and previously unseen patterns through statistically grounded uncertainty estimates, thereby enabling robust detection of novel classes and anomalous configurations in complex visual data.

A central contribution of this work lies in demonstrating how CP, when applied to the outputs of deep neural networks, supports both decision reliability and explainability.

Beyond improving classification robustness, CP provides an error-based signaling mechanism that reveals divergent or counterfactual patterns, offering insight into the model’s concept boundaries and generalization behavior.

In the context of Kandinsky patterns, this facilitates a deeper understanding of abstract concept learning and highlights the relevance of CP as a methodological building block for Explainable Artificial Intelligence (XAI), particularly in settings where epistemic uncertainty and distributional shift are critical.

These findings open several forward-looking research directions at the intersection of CP, pattern recognition, and trustworthy AI. A key challenge is the development of scalable and computationally efficient CP frameworks suitable for real-time and resource-constrained environments, such as autonomous systems, clinical decision support, e.g., real-world medical imaging [43,44], and continuous environmental monitoring. At the methodological level, tighter integration of CP with contemporary XAI approaches, including counterfactual explanations, concept-based explanations, and causality-aware models, could yield richer interpretability frameworks that are sensitive to domain-specific requirements. Extending CP beyond classical vision tasks to non-traditional and emerging domains represents another promising avenue. Applications to natural language processing, multimodal learning, and foundation models, particularly in combination with transformer architectures, could provide calibrated uncertainty estimates for language understanding, reasoning, and cross-modal fusion. Similarly, advancing CP-based counterfactual detection in high-dimensional, noisy, or weakly labeled data remains an open research challenge, with implications for anomaly detection, open-set recognition, and scientific discovery.

Future work should also explore CP in generalized few-shot and continual learning scenarios, where the reliable identification of novel classes from minimal data is essential. Integration with advanced neural architectures, including graph neural networks, vision transformers, and hybrid neuro-symbolic models, may further enhance both predictive performance and interpretability. From a human-centered perspective, visualizing conformal predictive distributions offers a pathway to align CP’s formal statistical guarantees with intuitive user interfaces, supporting transparency, trust calibration, and effective human–AI collaboration. Finally, systematic benchmarking, for example using synthetic testbeds such as Kandinsky patterns, is needed to standardize the evaluation of explainability, robustness, and counterfactual reasoning. In parallel, the role of CP in supporting fairness, privacy preservation, and accountability should be examined in light of emerging regulatory and ethical frameworks, positioning CP as a core methodology for robust, interpretable, and trustworthy AI systems.