Conditional Entropy-Based Sequential Decision-Making for AI Adoption in Manufacturing: A Reinforcement Learning Approach

Abstract

Most small- and medium-sized manufacturers face challenges in adopting artificial intelligence (AI) in production systems due to limited domain expertise and challenges in making interrelated decisions. This decision-making process can be characterized as sequential decision-making (SDM), in which guidance on the decision order is valuable. This study proposes a data-driven SDM framework to identify an effective order of key decision elements for AI adoption, aiming to rapidly reduce uncertainty at each decision stage. The framework employs a Q-learning-based reinforcement learning approach, using conditional entropy as the reward function to quantify uncertainty. Based on a review of 55 studies applying AI to milling processes, the proposed model identifies the following decision order that minimizes cumulative uncertainty: sensor, data collection interval, data dimension, AI technique, data type, and data collection period. To validate the model, we conduct simulations of 4000 SDM episodes under rule-based constraints using the number of corrected episodes as a performance metric. Simulation results show that the proposed model generates decision orders with no corrections and that knowing the relative order between two elements is more effective than knowing exact positions. The proposed data-driven framework is broadly applicable and can be extended to AI adoption in other manufacturing domains.

1. Introduction

As the manufacturing industry faces increasing pressure to remain competitive amid technological advances, artificial intelligence (AI) is emerging as a key enabler of operational innovation and efficiency [1,2]. In particular, AI directly enhances flexibility and productivity within manufacturing processes, accelerating business growth [1,3]. However, the actual application of AI in manufacturing and its resulting process improvements are progressing more slowly than expected [4]. One of the possible reasons for this delay is that most manufacturers are small- and medium-sized enterprises (SMEs); for example, over 98% of manufacturers in Europe and the U.S. are SMEs [5,6]. Small- and medium-sized manufacturers (SMMs) commonly lack in-house AI expertise [7]. In many SMMs, senior executives—who are not AI experts—are often the ones making decisions regarding AI adoption. This practice not only slows down the decision-making process but also creates barriers to adoption itself [1]. Given that SMMs account for the overwhelming majority of manufacturers, SMMs’ successful AI-driven transformation is essential for advancing the manufacturing industry as a whole. Supporting SMMs to make more efficient AI adoption decisions is therefore critical to the future of manufacturing. While it might be theoretically possible to provide AI-related human and technical resources to every SMM in the form of governmental funding, such an approach is unrealistic. Alternatively, a more feasible and impactful strategy is to develop a decision support system tailored to the needs of SMM executives, the decision-makers. Especially, such a system can enable decision-makers without AI-related domain knowledge to effectively and efficiently evaluate key aspects of AI adoption and make appropriate decisions. Such a decision support system can also help capital-constrained SMMs adopt AI readily without hiring AI professionals [1,8].

Decision-making for AI adoption in manufacturing can be described as a form of sequential decision-making (SDM). SDM refers to the approach of making decisions step by step, particularly when multiple interrelated elements and their attributes must be determined [9,10]. In this context, elements are defined as distinct decision points that collectively comprise the overall decision-making process. For example, the decision-making process for AI adoption in manufacturing may include various elements such as sensor, data collection interval, and data type. Attributes refer to candidate options that can be selected for each element. For instance, the sensor attribute may include cameras, energy meters, and force sensors. Meanwhile, research studies systematically examining SDM problems regarding the AI-adopting process remain sparse. Although various SDM approaches have been applied in fields such as robotics, games, and control tasks, these studies typically focus on temporal action sequences [11]. However, such approaches are not well-suited for AI adoption in manufacturing, which requires handling logical interdependencies between elements. These logical interdependencies become evident when the selection of one element constrains the feasible choices for others. An effective and efficient way to address SDM problems is to determine an appropriate decision order that reduces uncertainty and narrows down the range of options early [10]. Although uncertainty reduction is one of the promising approaches for addressing SDM problems, it has rarely been explored systematically in prior studies. Thus, there is a clear need for a new approach that incorporates uncertainty reduction into the element ordering in SDM.

The strategy to set the budget first in a car-purchasing decision represents a typical case of SDM aimed at reducing uncertainty. In SDM, there is typically no predefined order of decision-making elements. Thus, the number of available choices is extensive. However, if the budget is specified first, the range of feasible brands, sizes, and models becomes significantly narrower. As such, once the decision order is defined, the number of combinations to be explored (i.e., the uncertainty) is significantly reduced, making the overall SDM process more efficient. This reduction in uncertainty becomes particularly important when the attributes within decision-making elements are interdependent [12]. Considering such interdependencies allows for a more informed decision order that minimizes uncertainty at each stage. Here, a stage denotes a single step in the SDM process during which one element is determined. The following example illustrates how interdependencies among elements affect the decision order in the SDM process for AI adoption in manufacturing. Selecting an energy meter (an attribute) as the sensor (an element) implicitly fixes the data type (another element) as numeric data (an attribute). In contrast, starting with the data type as numeric data may still allow for a wide range of sensor options, including energy, force, position, and pressure meters. Meanwhile, such interdependencies are also a kind of domain knowledge. Thus, decision-makers without domain knowledge can struggle to catch interdependencies among elements and determine appropriate decision orders. Therefore, defining an appropriate decision-making order can reduce the overall decision complexity and enhance the efficiency of an SDM support system. Accordingly, this study proposes a systematic, data-driven approach that provides guidelines for SDM in AI adoption for manufacturing, taking into account the interdependencies and uncertainties inherent in SDM. In particular, this study emphasizes minimizing the total uncertainty that decision-makers face throughout the SDM process by identifying a decision order that maximizes uncertainty reduction at each stage.

Conditional entropy (CE), a fundamental concept in information theory, measures the amount of uncertainty remaining about a random variable Y given that another variable X is known. It is a useful tool for quantifying how much information is gained when one variable is revealed [13]. By applying this concept to SDM, we can evaluate the uncertainty reduction at each decision point. In addition, based on this idea, we develop an algorithm that identifies the efficient order of SDM elements that yields the maximized reduction in CE at each stage. To implement this, we propose a reinforcement learning (RL) model that uses CE as the reward function within the Q-learning framework. RL is well-suited for solving decision-making problems involving multiple interdependent elements [14]. Among various RL algorithms, Q-learning is particularly suitable due to its broad applicability [15]. However, the integration of CE and Q-learning for solving SDM problems has been largely unexplored, highlighting the need to fill this research gap.

This study aims to develop a data-driven SDM model, built upon an extensive dataset of real AI adoption cases, that helps SMMs determine the efficient order of decision elements for AI adoption in manufacturing. To this end, we investigate how uncertainty can be quantified and minimized through the integration of information-theoretic metrics and RL within the SDM framework. Then, we validate the proposed SDM model through simulation to assess its practical effectiveness, since simulation provides a controlled environment to test the SDM under realistic constraints. Through this approach, this study contributes to structured decision-making in AI adoption by proposing a novel SDM model and offering practical support for overcoming the challenges that SMMs face. Furthermore, the proposed framework is designed to be scalable and policy-relevant, providing a foundation for broader applications of structured SDM across diverse domains.

2. Literature Review

This section reviews various approaches to solving SDM problems. This section also defines SDM’s elements and their corresponding attributes based on a review of 55 studies on AI adoption in manufacturing. These 55 studies also serve as training data for the proposed model. In addition, RL and the Q-learning algorithm are reviewed to be utilized in the development of the SDM model.

2.1. Sequential Decision-Making

SDM problems have been widely studied in various domains such as robotics, finance, and healthcare [11,16]. However, the goal of existing SDM studies is to select a sequence of actions to maximize external task rewards. For this reason, most prior studies focus on the temporal dependency of sequential action decisions, rather than considering the interdependencies or uncertainties among SDM elements.

Chen et al. (2021) addressed an SDM problem in which actions unfold in a temporal sequence, focusing on diverse environments such as games and robotic control [17]. More specifically, the proposed model was trained to select an initial action that leads to a sequence of subsequent actions yielding the highest overall reward. In this setting, earlier actions help predict and shape later actions through temporal dependencies. In contrast, SDM for AI adoption in manufacturing is characterized by logical dependencies because the selection of one decision element constrains the feasible options for others. Such an SDM problem can be solved not with temporal progression, but with how decisions reduce uncertainty in the decision structure.

Lee et al. (2022) addressed an SDM problem involving the selection of appropriate actions in various game environments [18]. Their approach focused on learning temporal patterns from observed trajectories to generate high-performing sequences that maximize cumulative rewards. In this approach, the SDM model predicts each next event by relying on the learned relationships between earlier and later events in the trajectory, emphasizing temporal continuity. However, SDM for AI adoption in manufacturing does not follow a fixed temporal sequence but rather involves selecting interdependent decision elements. In such structured decision spaces, the key challenge lies in determining which decision reduces the overall uncertainty in the SDM process, where logical constraints exist between elements, rather than simply predicting the next step in a temporal sequence.

Janner et al. (2021) solved an SDM problem focused on reproducing high-quality decision sequences through imitation, particularly in tasks such as robotic manipulation and navigation [19]. Their method aimed to generate successful behavior sequences by imitating trajectories from prior examples, using a diffusion-based process that gradually refines random inputs. More specifically, the model learns to produce sequences that resemble expert demonstrations, relying on observed action chains without explicitly modeling decision dependencies. In contrast, SDM in AI adoption for manufacturing requires not imitation but deliberate structuring of decision order, in which the selection of one element logically determines or limits the available attributes for other elements to be decided. Thus, the focus is on managing structural interdependencies and uncertainty rather than on recreating previously successful patterns.

SDM problems have also been examined in autonomous driving, focusing on sequential maneuvers in dynamic traffic environments [20]. In the energy domain, hierarchical RL has been applied to support the SDM process in regional market trading temporal processes [21]. In agriculture, SDM concepts have been explored in computer vision tasks such as real-time poultry monitoring [22]. Human-in-the-loop learning has also been studied as an SDM setting, with approaches such as RL from feedback and preference-based learning [23]. These studies, however, primarily emphasize temporal decision sequences and do not explicitly address the structural interdependencies among decision elements.

Prior SDM studies have primarily focused on predicting sequences of actions based on past trajectories, treating each action as a time-indexed step toward maximizing a cumulative reward. However, solving the SDM problem of AI adoption in manufacturing needs to focus on the efficient order of structured decision elements, in which each choice constrains the space of subsequent options. In this setting, quantifying uncertainty reduced at each step can be critical to identifying a decision order. Therefore, the methodologies and approaches proposed in existing studies are not suitable for modeling SDM for AI adoption in manufacturing. Instead, information-theoretic methods can be more effectively utilized in this context, as they allow for quantifying the uncertainty.

2.2. Decision-Making Based on Information Theory

Information-theoretic approaches are widely used for analyzing decision-making structures and solving related problems. Information theory, introduced by Claude Shannon in 1948, provides a mathematical framework for quantifying uncertainty and measuring the amount of information to design more efficient decision-making processes [13,24]. In information theory, uncertainty is numerically expressed through the concept of entropy. Entropy quantifies the degree of uncertainty associated with a random variable and is calculated as shown in Equation (1) [13,24]. Entropy $H\left(X\right)$ of a discrete random variable $X$ is calculated based on the probability $p\left(x\right)$ of each possible value $x$ that $X$ can take. A higher entropy value $H\left(X\right)$ indicates a greater level of uncertainty and a more dispersed distribution of information.

$$H\left(X\right)=-\sum _{x\in X}p\left(x\right){\mathit{log}}_{2}p\left(x\right)$$

(1)

In decision-making problems, various entropy-derived information-theoretic metrics—such as mutual information (MI), total correlation (TC), and CE—are commonly used. MI represents the amount of information shared between two random variables. MI between random variables $X$ and $Y$ is calculated based on the joint probability $p\left(x,y\right)$ of the simultaneous occurrence of two events, along with the marginal probabilities $p\left(x\right)$ and $p\left(y\right)$, as shown in Equation (2) [13,24]. When the MI value $I\left(X;Y\right)$ is zero, it indicates that $X$ and $Y$ are statistically independent [24]. MI measures how much two random variables are informationally connected—that is, it quantifies the degree of dependence between them.

$$I\left(X;Y\right)=\sum _{x\in X}\sum _{y\in Y}p\left(x,y\right){\mathit{log}}_2\left({\displaystyle \frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}}\right)$$

(2)

Since MI is defined for a pair of variables, other metrics such as TC are used to analyze informational dependencies among three or more variables. TC measures the shared information (or redundancy) among all variables in a given set [24]. For a set of random variables $\{X_1,\dots ,X_m\}$, the TC value $C(X_1;\dots \hspace{0.17em};X_m)$ is defined as shown in Equation (3) [24]. In this equation, $H(X_1,\dots ,X_m)$ represents joint entropy of all variables, and it is calculated as Equation (4).

$$C\left(X_1;\dots ;X_m\right)=\sum _{i=1}^{m}H\left(X_i\right)-H(X_1,\dots ,X_m)$$

(3)

$$H\left(X_1,\dots ,X_m\right)=-\sum _{x_1\in X_1}\dots \sum _{x_m\in X_m}p\left(x_1,\dots ,x_m\right){\mathit{log}}_{2}p\left(x_1,\dots ,x_m\right)$$

(4)

Since MI and TC measure the amount of redundant information shared among variables, it is commonly used for variable selection (i.e., choosing the most relevant input features) or dependency analysis (i.e., identifying how strongly variables are related to each other). Qian and Shu proposed an MI-based variable selection algorithm aiming to reduce redundancy and improve classification performance through a forward greedy search strategy [25]. Todorov and Setchi proposed an MI-based variable selection method for classification tasks using discrete data [26]. Qiu and Niu proposed a variable selection method based on TC-based information metric to reduce redundancy among variables, aiming to improve classification performance in high-dimensional data [27]. Li et al. compared MI and TC to quantify the multivariate dependency among neural signals, enabling the analysis of functional connectivity across brain regions [28]. Collectively, previous studies based on MI and TC have primarily focused on quantifying the redundancy or relevance among variables. While MI and TC are well-suited for global variable selection problems, they are not suitable for solving SDM problems.

One promising approach to solving the SDM problem is to minimize the remaining set of options (i.e., uncertainty) at each stage. In such cases, CE can serve as a useful measure for guiding the decision-making process. CE of a random variable $Y$ given another random variable $X$ is quantified by the expression in Equation (5). CE measures the uncertainty that remains about $Y$ when $X$ is known [13]. A CE value of zero indicates that $Y$ is statistically dependent on $X$; that is, knowing $X$ completely eliminates the uncertainty in $Y$ [24]. Conversely, CE reaches its maximum when $X$ and $Y$ are statistically independent [24].

$$H\left(Y|X\right)=-\sum _{x\in X,y\in Y}p\left(x,y\right){\mathit{log}}_{2}p(y|x)$$

(5)

Although these characteristics of CE make it well-suited for solving SDM problems, a very limited number of studies have directly applied CE to decision-making problems. Dai et al. proposed a CE-based attribute selection (i.e., removing less important attributes to simplify decision-making) method for incomplete decision systems, aiming to reduce uncertainty in classification [29]. Yang et al. proposed a CE-based entropy model, aiming to quantify uncertainty in fuzzy decision information [30].

Overall, information-theoretic approaches have been widely applied to decision-making problems to quantify uncertainty, identify relevant variables, and analyze informational dependencies. As summarized in

Table 1. Summary of literature review on decision-making based on information theory.

Reference	Used Metric	Purpose	Problem Type
[25]	MI	Variable selection	Classification accuracy improvement
[26]	MI	Variable selection	Classification accuracy improvement
[27]	TC	Variable selection	Classification accuracy improvement
[28]	MI and TC	Dependency analysis	Functional connectivity analysis
[29]	CE	Attribute selection	Classification accuracy improvement
[30]	CE	Uncertainty quantification	Fuzzy decision modeling
This study	CE	Uncertainty reduction	Sequential decision-making

, most prior studies have focused on variable selection or reducing redundancy to improve classification accuracy using metrics such as MI and TC. These metrics have proven to be effective for global variable selection or dependency analysis, in which the overall structure of relationships among variables is of primary concern.

Table 2. Key properties of MI, TC, and CE.

Metric	Applicable Number of Variables	Measured Information	Dependency Type
MI	2	Shared information	Mutual
TC	≥2	Redundant information	Joint
CE	≥2	Uncertainty	Conditional

summarizes the key properties of MI, TC, and CE. To begin with, MI is not appropriate to deal with three or more variables. Moreover, the most critical point is that MI and TC are fundamentally used to assess the amount of shared or redundant information across multiple variables. However, solving SDM problems that aim to minimize remaining uncertainty at each decision stage requires a different perspective. In this context, the ability to quantify uncertainty under conditional circumstances is critical, as decisions at later stages depend on those made earlier. In such cases, CE serves as a more appropriate metric, as it measures the remaining uncertainty of a target variable given a known condition. Despite such suitability of CE, no known studies have directly applied CE to SDM structures. Therefore, to address this research gap, this study aims to propose a CE-based SDM model. Specifically, this model minimizes CE at each stage to support more efficient decision-making processes under multi-stage decision structures.

2.3. Sequential Decision-Making Elements and Attributes for AI Adoption in Manufacturing

We review 55 previous studies on AI adoption in manufacturing to identify the key decision-making elements and the corresponding attributes of each element (see

Table A1. Literature on AI adoption in manufacturing.

No.	Reference No.	Reference	Purpose	Task	Sensor	Data Type	Data Collection Interval	Data Collection Period	Data Dimension	AI Technique
1	1	[44]	Quality	Diagnosis	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	SVM
2	2	[45]	Quality	Diagnosis	Force	Numeric	<0.01 s	Unknown	<10 (1D)	SVM
3			Quality	Diagnosis	Force	Numeric	<0.01 s	Unknown	<10 (1D)	Others
4			Quality	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	SVM
5			Quality	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	Others
6			Energy	Diagnosis	Force	Numeric	<0.01 s	Unknown	<10 (1D)	SVM
7			Energy	Diagnosis	Force	Numeric	<0.01 s	Unknown	<10 (1D)	Others
8			Energy	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	SVM
9			Energy	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	Others
10	3	[46]	Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
11			Quality	Prediction	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
12			Quality	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
13			Fault	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
14			Fault	Monitoring	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
15			Fault	Monitoring	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
16			Quality	Prediction	Camera	Image	Unknown	Unknown	Unknown	NN
17			Fault	Monitoring	Camera	Image	Unknown	Unknown	Unknown	NN
18	4	[47]	Cost	Control/Optimization	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	SVM
19			Cost	Control/Optimization	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	NN
20			Quality	Diagnosis	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	SVM
21			Quality	Diagnosis	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	NN
22			Quality	Control/Optimization	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	SVM
23			Quality	Control/Optimization	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	NN
24			Energy	Diagnosis	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	SVM
25			Energy	Diagnosis	Others	Numeric	<0.01 s	Unknown	10~100 (1D)	NN
26	5	[48]	Cost	Diagnosis	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
27			Cost	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
28	6	[49]	Cost	Prediction	Camera	Image	0.01~1 s	1~7days	>100 × 100 (2D)	NN
29			Cost	Control/Optimization	Camera	Image	0.01~1 s	1~7days	>100 × 100 (2D)	NN
30			Quality	Prediction	Camera	Image	0.01~1 s	1~7days	>100 × 100 (2D)	NN
31			Quality	Control/Optimization	Camera	Image	0.01~1 s	1~7days	>100 × 100 (2D)	NN
32	7	[50]	Cost	Control/Optimization	Force	Numeric	Unknown	Unknown	10~100 (1D)	SVM
33			Cost	Control/Optimization	Force	Numeric	Unknown	Unknown	10~100 (1D)	Others
34			Energy	Prediction	Force	Numeric	Unknown	Unknown	10~100 (1D)	SVM
35			Energy	Prediction	Force	Numeric	Unknown	Unknown	10~100 (1D)	Others
36			Energy	Control/Optimization	Force	Numeric	Unknown	Unknown	10~100 (1D)	SVM
37			Energy	Control/Optimization	Force	Numeric	Unknown	Unknown	10~100 (1D)	Others
38	8	[51]	Cost	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
39			Cost	Prediction	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
40			Cost	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
41			Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
42			Quality	Prediction	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
43			Quality	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
44			Energy	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
45			Energy	Prediction	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
46			Energy	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
47	9	[52]	Cost	Monitoring	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
48			Cost	Diagnosis	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
49			Quality	Monitoring	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
50			Quality	Diagnosis	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
51	10	[53]	Cost	Diagnosis	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
52			Cost	Diagnosis	Sound	Numeric	Unknown	Unknown	<10 (1D)	SVM
53			Time	Diagnosis	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
54			Time	Diagnosis	Sound	Numeric	Unknown	Unknown	<10 (1D)	SVM
55	11	[54]	Quality	Diagnosis	Force	Numeric	Unknown	Unknown	10~100 (1D)	NN
56			Quality	Diagnosis	Vibration	Numeric	Unknown	Unknown	10~100 (1D)	NN
57			Energy	Diagnosis	Force	Numeric	Unknown	Unknown	10~100 (1D)	NN
58			Energy	Diagnosis	Vibration	Numeric	Unknown	Unknown	10~100 (1D)	NN
59	12	[55]	Cost	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
60			Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
61	13	[56]	Cost	Monitoring	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
62			Time	Monitoring	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
63	14	[57]	Quality	Monitoring	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	NN
64			Quality	Prediction	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	NN
65	15	[58]	Quality	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	SVM
66			Quality	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	Others
67	16	[59]	Time	Prediction	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
68			Time	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
69			Quality	Prediction	Sound	Numeric	<0.01 s	Unknown	<10 (1D)	NN
70			Quality	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
71	17	[60]	Quality	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
72			Quality	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	Others
73			Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
74			Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	Others
75	18	[61]	Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
76			Time	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
77	19	[62]	Quality	Prediction	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
78			Quality	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
79			Energy	Prediction	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
80			Energy	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
81	20	[63]	Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
82			Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
83			Cost	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
84			Cost	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
85			Time	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
86			Time	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
87			Time	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
88			Time	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
89	21	[64]	Quality	Prediction	Force	Numeric	<0.01 s	<1 day	<10 (1D)	Tree
90			Quality	Prediction	Force	Numeric	<0.01 s	<1 day	<10 (1D)	Others
91	22	[65]	Time	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
92			Time	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
93			Energy	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
94			Energy	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
95			Time	Prediction	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	SVM
96			Time	Control/Optimization	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	SVM
97			Energy	Prediction	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	SVM
98			Energy	Control/Optimization	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	SVM
99	23	[66]	Quality	Monitoring	Velocity/Acceleration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
100			Quality	Monitoring	Velocity/Acceleration	Numeric	<0.01 s	Unknown	<10 (1D)	Others
101	24	[67]	Quality	Prediction	Force	Numeric	<0.01 s	Unknown	10~100 (1D)	NN
102			Quality	Prediction	Vibration	Numeric	0.01~1 s	Unknown	10~100 (1D)	NN
103			Quality	Control/Optimization	Force	Numeric	<0.01 s	Unknown	10~100 (1D)	NN
104			Quality	Control/Optimization	Vibration	Numeric	0.01~1 s	Unknown	10~100 (1D)	NN
105	25	[68]	Quality	Diagnosis	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	NN
106			Quality	Control/Optimization	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	NN
107	26	[69]	Cost	Prediction	Sound	Numeric	<0.01 s	<1 day	10~100 (1D)	NN
108			Cost	Prediction	Sound	Numeric	<0.01 s	<1 day	10~100 (1D)	Others
109			Time	Prediction	Sound	Numeric	<0.01 s	<1 day	10~100 (1D)	NN
110			Time	Prediction	Sound	Numeric	<0.01 s	<1 day	10~100 (1D)	Others
111	27	[70]	Time	Prediction	Energy	Numeric	<0.01 s	<1 day	<10 (1D)	NN
112			Cost	Prediction	Energy	Numeric	<0.01 s	<1 day	<10 (1D)	NN
113	28	[71]	Quality	Prediction	Force	Numeric	0.01~1 s	Unknown	<10 (1D)	NN
114	29	[72]	Quality	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
115			Energy	Diagnosis	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
116	30	[73]	Cost	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
117			Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
118	31	[74]	Cost	Monitoring	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
119			Quality	Monitoring	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
120			Energy	Monitoring	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
121	32	[75]	Quality	Monitoring	Others	String	Unknown	Unknown	<10 (1D)	NN
122			Quality	Prediction	Others	String	Unknown	Unknown	<10 (1D)	NN
123			Quality	Monitoring	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
124			Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
125	33	[76]	Cost	Monitoring	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
126			Cost	Prediction	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
127			Quality	Monitoring	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
128			Quality	Prediction	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
129	34	[77]	Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	Tree
130			Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	kNN
131			Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
132			Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	Others
133			Energy	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	Tree
134			Energy	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	kNN
135			Energy	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
136			Energy	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	Others
137	35	[78]	Quality	Diagnosis	Energy	Numeric	<0.01 s	Unknown	<10 (1D)	NN
138	36	[79]	Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
139			Quality	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
140	37	[80]	Quality	Prediction	Velocity/Acceleration	Numeric	Unknown	Unknown	<10 (1D)	NN
141	38	[81]	Cost	Diagnosis	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	NN
142	39	[82]	Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
143			Cost	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	Others
144			Time	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
145			Time	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	Others
146	40	[83]	Cost	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
147			Cost	Monitoring	Velocity/Acceleration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
148			Quality	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
149			Quality	Monitoring	Velocity/Acceleration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
150			Energy	Monitoring	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
151			Energy	Monitoring	Velocity/Acceleration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
152	41	[84]	Quality	Diagnosis	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	NN
153			Fault	Diagnosis	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	NN
154	42	[85]	Cost	Control/Optimization	Energy	Numeric	Unknown	Unknown	<10 (1D)	SVM
155			Cost	Control/Optimization	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
156			Cost	Control/Optimization	Force	Numeric	Unknown	Unknown	<10 (1D)	SVM
157			Cost	Control/Optimization	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
158			Cost	Control/Optimization	Sound	Numeric	Unknown	Unknown	<10 (1D)	SVM
159			Cost	Control/Optimization	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
160			Cost	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
161			Cost	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
162			Time	Control/Optimization	Energy	Numeric	Unknown	Unknown	<10 (1D)	SVM
163			Time	Control/Optimization	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
164			Time	Control/Optimization	Force	Numeric	Unknown	Unknown	<10 (1D)	SVM
165			Time	Control/Optimization	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
166			Time	Control/Optimization	Sound	Numeric	Unknown	Unknown	<10 (1D)	SVM
167			Time	Control/Optimization	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
168			Time	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
169			Time	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
170			Fault	Prediction	Energy	Numeric	Unknown	Unknown	<10 (1D)	SVM
171			Fault	Prediction	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
172			Fault	Prediction	Force	Numeric	Unknown	Unknown	<10 (1D)	SVM
173			Fault	Prediction	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
174			Fault	Prediction	Sound	Numeric	Unknown	Unknown	<10 (1D)	SVM
175			Fault	Prediction	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
176			Fault	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
177			Fault	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
178			Fault	Control/Optimization	Energy	Numeric	Unknown	Unknown	<10 (1D)	SVM
179			Fault	Control/Optimization	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
180			Fault	Control/Optimization	Force	Numeric	Unknown	Unknown	<10 (1D)	SVM
181			Fault	Control/Optimization	Force	Numeric	Unknown	Unknown	<10 (1D)	NN
182			Fault	Control/Optimization	Sound	Numeric	Unknown	Unknown	<10 (1D)	SVM
183			Fault	Control/Optimization	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
184			Fault	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	SVM
185			Fault	Control/Optimization	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
186	43	[86]	Quality	Monitoring	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	Tree
187	44	[87]	Fault	Prediction	Force	Numeric	<0.01 s	<1 day	<10 (1D)	Tree
188	45	[88]	Quality	Monitoring	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
189			Quality	Monitoring	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
190			Quality	Monitoring	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
191			Fault	Monitoring	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
192			Fault	Monitoring	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
193			Fault	Monitoring	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
194			Fault	Prediction	Energy	Numeric	Unknown	Unknown	<10 (1D)	NN
195			Fault	Prediction	Sound	Numeric	Unknown	Unknown	<10 (1D)	NN
196			Fault	Prediction	Vibration	Numeric	Unknown	Unknown	<10 (1D)	NN
197	46	[89]	Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
198			Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
199			Quality	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
200			Quality	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
201	47	[90]	Quality	Monitoring	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
202			Quality	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
203	48	[91]	Quality	Prediction	Camera	Image	Unknown	Unknown	>100 × 100 (2D)	NN
204	49	[92]	Fault	Diagnosis	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
205	50	[93]	Quality	Prediction	Force	Numeric	<0.01 s	Unknown	<10 (1D)	NN
206			Quality	Prediction	Vibration	Numeric	<0.01 s	Unknown	<10 (1D)	NN
207	51	[94]	Energy	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
208			Energy	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	SVM
209	52	[95]	Quality	Prediction	Others	String	Unknown	Unknown	<10 (1D)	NN
210			Quality	Control/Optimization	Others	String	Unknown	Unknown	<10 (1D)	NN
211			Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
212			Quality	Control/Optimization	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
213	53	[96]	Quality	Monitoring	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	NN
214			Quality	Prediction	Vibration	Numeric	<0.01 s	<1 day	<10 (1D)	NN
215	54	[97]	Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
216			Fault	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN
217	55	[98]	Quality	Prediction	Others	Numeric	Unknown	Unknown	<10 (1D)	NN

in Appendix A). In particular, the review focuses on the machining process, as it is one of the most commonly applied, representative processes in manufacturing [31]. Furthermore, to obtain sufficient information for decision-making model development, we select studies that clearly describe the data collection methods, the AI technique applied, and the objectives of AI adoption. According to the elements commonly identified across the 55 reviewed studies, we derive eight decision-making elements: purpose, task, sensor, data type, data collection interval, data collection period, data dimension, and AI technique. In addition, we list the corresponding attributes for each element in

Table 3. Decision-making elements and attributes for AI adoption in manufacturing.

Elements	Attributes
Purpose	Cost, Energy, Fault, Time, Quality
Task	Control/Optimization, Diagnosis, Monitoring, Prediction
Sensor	Camera, Energy, Force, Position, Pressure, Rotation, Sound, Velocity/Acceleration, Vibration, Others
Data type	Image, Numeric, String
Data collection interval	<0.01 s, 0.01~1 s, >1 s
Data collection period	<1 day, 1 day~7 days, 7 days~1 month, >1 month
Data dimension	<10 (1D), 10~100 (1D), >100 (1D), <100 × 100 (2D), >100 × 100 (2D)
AI technique	Clustering, kNN, Linear, NN, SVM, Tree, Others

.

Adopting AI in manufacturing inevitably involves both a purpose and a task. For example, AI can be introduced to predict energy consumption, diagnose faults, or optimize production time. Since AI relies on data, implementing it involves key decisions regarding how data will be collected, such as sensor selection, data type, and collection parameters. These data-related decision-making elements include the sensor, data type, data collection interval, data collection period, and data dimension.

Sensors can be selected from various attributes such as cameras, energy meters, and force sensors. The data collected from these sensors can take the data type of images, numerical values, or strings. The data collection interval and data collection period are typically determined at the discretion of the user.

Data dimension can be broadly categorized into one-dimensional (1D) and two-dimensional (2D) types, and each can have several attributes depending on scale and resolution. In this study, we define 1D data as single vector-type data that varies over time or sequence (e.g., sound, text, and vibration), and 2D data as matrix-type data with spatial or visual structures (e.g., images and position grids). The data dimension in this study refers to the raw sensor-level data and does not consider the transformed input structures that can be used later for AI models.

Finally, the AI technique element includes widely used methods, such as clustering, k-nearest neighbor (kNN), neural networks (NNs), and support vector machines (SVMs). Attributes that do not fall into predefined categories are grouped under ‘others’.

In addition, based on the same 55 studies reviewed to identify decision-making elements and their attributes, we extract a total of 217 decision cases for the development of the data-driven SDM model (see Table A1 in Appendix A). These 217 decision cases describe how the attributes of each element were selected in actual AI adoption scenarios. These decision cases are then converted into categorical variables and used to calculate CE among the decision-making elements within the SDM model.

Table 4. Categorical variables data example used for model development.

No.	Purpose	Task	Sensor	Data Type	Data Collection Interval	Data Collection Period	Data Dimension	AI Technique
1	5	2	9	2	1	1	1	5
2	5	2	3	2	1	0	1	5
3	5	2	3	2	1	0	1	7
4	5	2	9	2	1	0	1	5
5	5	2	9	2	1	0	1	7

illustrates examples of the data, encoded as categorical variables, that are utilized in the development of the model.

2.4. Reinforcement Learning and Q-Learning for Sequential Decision-Making

Several approaches have been proposed to model situations in which elements are sequentially selected based on entropy reduction or information gain. For example, decision-tree algorithms sequentially partition attributes based on information gain (i.e., entropy reduction). However, decision-tree algorithms generally evaluate local uncertainty reduction at each split and cannot explicitly account for how earlier choices affect the uncertainty of later ones [32,33].

As reviewed in Section 2.3, decision-making on AI adoption in manufacturing involves eight interdependent decision elements, each with three to ten possible attributes. That is, this SDM problem is significantly complex, and each choice influences the others. Solving complex SDM problems can be formulated as a Markov decision process (MDP) [34]. MDP provides a mathematical framework for modeling decision-making in situations where outcomes are partly random and partly under the control of decision-makers [35]. More specifically, an MDP calculates the expected utility of each possible action in a given state [35]. However, theoretical approaches using MDPs often face practical limitations due to the computational cost of constructing a transition probability matrix (TPM) for all possible state-action pairs; this issue is commonly termed the curse of dimensionality [34]. To deal with this challenge, RL offers an alternative approach that does not require explicit modeling of TPM. Instead, RL enables the agent to learn optimal decision policies through interaction with the environment [14].

Several prior studies have applied entropy-based measures within RL frameworks to address SDM problems. For example, Filippi et al. (2010) proposed an RL algorithm using Kullback–Leibler divergence, an entropy-derived metric, mathematically addressing the trade-off between exploration and exploitation [36]. Li et al. (2021) introduced an RL framework for active feature acquisition, aiming to balance predictive accuracy with acquisition costs by evaluating feature redundancy using MI [37]. Similarly, Maliah and Shani (2018) applied RL to model cost-sensitive classification as an MDP, incorporating entropy-based split criteria commonly used in decision trees [38]. While these studies demonstrated the usefulness of entropy-related metrics within RL frameworks, their primary objectives are to improve prediction accuracy or to reduce costs. In addition, the prior approaches did not explicitly quantify logical interdependencies among decision elements or minimize overall uncertainty across sequential choices. Addressing this gap, our study employs CE to explicitly capture interdependencies among decision elements and to reduce cumulative uncertainty in the SDM process.

In RL, the agent continuously interacts with the environment in a feedback loop. At each time step $t$, the environment provides the agent with a state $s_t$, representing the current situation. In response to the state, the agent selects an action $a_t$, and the environment returns a reward $r_{t+1}$, a scalar signal indicating the immediate outcome of the chosen action. Over time, the agent aims to learn an optimal policy that maximizes the cumulative reward through experience and policy updates [14].

A central concept in RL is the Q-value, which estimates the expected return for an agent taking action $a$ in state $s$ and subsequently following a policy $\pi$. The Q-value $Q^{\pi }(s,a)$ is defined based on the Bellman expectation equation, as shown in Equation (6) [14,39].

$$Q^{\pi }\left(s,a\right)=E_{\pi }\left[\left.\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}\right|s_t=s, a_t=a\right]$$

(6)

Here, $\gamma$ is the discount factor, which determines the present value of future rewards, and the expectation $E_{\pi }$ is taken over all possible future trajectories generated by policy $\pi$. This formulation is used to evaluate a given policy. To find the optimal policy ${\pi }^{*}$, the Bellman optimality equation defines $Q^{*}\left(s,a\right)$ as the maximum expected return over all possible next actions, as shown in Equation (7) [14,39].

$$Q^{*}\left(s,a\right)=E\left[\left.r_{t+1}+\gamma \underset{a^{\prime }}{\mathit{max}}{Q}^{*}\left(s_{t+1},a^{\prime }\right)\right|s_t=s, a_t=a\right]$$

(7)

In this equation, $a^{\prime }$ denotes a possible action available in the next state $s_{t+1}$, and the maximization identifies the optimal action with the highest expected return [14].

Among various RL algorithms, Q-learning is one of the most commonly used algorithms due to its applicability and simplicity [15]. Q-learning iteratively updates the Q-values at each step, allowing the agent to learn to select the action with the highest Q-value in each state [15]. One notable advantage of Q-learning is its ability to directly reflect the objective of decision-making within the reward function. For example, Khriji et al. defined a reward function based on route suitability to determine the optimal path for a mobile robot [40]. Similarly, Chakole et al. formulated a reward function using profits and losses to develop optimal trading strategies in the stock market [41].

This study aims to develop an SDM model leveraging the characteristics of the Q-learning algorithm. Specifically, at each decision-making stage (state), the policy determines which element to select (action), and the reward function is defined by the change in CE resulting from that selection. By using CE within the reward function, the model learns an efficient order of decisions that minimizes uncertainty throughout the sequence.

Although many prior studies have proposed methodologies for solving SDM problems, these approaches are not directly applicable to modeling SDM for AI adoption in manufacturing due to their primary focus on temporal dependencies. In addition, while many prior studies have extensively applied information-theoretic measures such as MI and TC for decision-making, these metrics are not well-suited for solving the SDM problems that aim at reducing uncertainty and determining efficient decision order. In such problems, CE is more appropriate. Nevertheless, CE has not been directly applied to SDM problems. Likewise, although RL and Q-learning have been widely used in complex decision-making scenarios, their integration with CE to support SDM remains underexplored. These observations reveal three key research gaps: (i) the lack of SDM methodologies tailored to AI adoption in manufacturing, (ii) the limited use of CE for uncertainty reduction in structured decision processes, and (iii) the absence of a combined CE and Q-learning approach for modeling SDM. To address these gaps, this study proposes a CE-based SDM model for AI adoption in manufacturing. The model uses Q-learning, with CE as the reward, to identify the efficient order of decision elements that minimizes the total amount of uncertainty in the entire SDM process. We define eight decision elements and their attributes by reviewing 55 prior studies and construct a dataset of 217 decision cases. We then validate the model through simulations under varying levels of prior knowledge. This study provides a scalable decision support approach, especially useful for SMMs with limited AI expertise. The proposed model also has the potential to support a wide range of SDM problems beyond manufacturing by providing a generalizable framework for structured, uncertainty-aware decision-making.

3. Proposed Sequential Decision-Making Model

To develop an SDM model, we design an RL model based on a Q-learning algorithm, as outlined in

Table 5. Nomenclature of symbols used in the RL-based SDM model.

Symbol	Description
$E$	Full set of decision-making elements
$e$	Single decision-making element in the set $E$
$S$	Set of elements already selected
$s$	Current state (i.e., $E\backslash S$)
$x$	Element selected as an action
$a$	Action (i.e., selecting decision element)
$r(x)$	Reward for selecting element $x$
$u_x$	Proportion of unknown attributes in element $x$
$H(e)$	Entropy of element $e$
$H\left(e|S\right)$	Conditional entropy of ee given selection set $S$

and

Table 6. Design of the RL-based SDM model.

State	$s=E\backslash S$
Action	$a=x\in E\backslash S$
Reward	$r(x)=\left[{\displaystyle \sum _{e\in E\backslash (S\cup \left\{x\right\})}}H\left(e|S\right)-H\left(e|S\cup \left\{x\right\}\right)\right]\cdot (1-u_x)$

, in which the key notations and the overall structure of the model are, respectively, summarized. Among the eight decision-making elements identified in Section 2.3, purpose and task are considered prerequisites for AI adoption in manufacturing. Without defining these two elements first, the subsequent decision-making process cannot proceed meaningfully. Moreover, determining purpose and task is not a domain knowledge-dependent decision-making. Therefore, we assume that purpose and task are predetermined, and the objective of the SDM model is to determine the efficient order for the remaining six elements.

In the formulations in Table 6, $E$ is the full set of candidate elements $e$, and $S$ denotes the set of elements that have already been selected. The state $s$ is defined as the set difference $E\backslash S$, i.e., the elements yet to be selected. Although this state is defined based on the milling process data used in this study, it is generalized to be extensible to SDM problems in other processes. The action $a$ refers to selecting one element $x\in E\backslash S$ in the current state. The reward function $r(x)$ measures the reduction in CE across the remaining elements $E\backslash (S\cup \{x\})$ when the candidate element $x$ is added to the selected set $S$. The reward is scaled by a penalty term $(1-u_x)$, where $u_x$ denotes the proportion of unknown attributes associated with element $x$. In this context, unknown attribute indicates that the applicable value for a given element is not identifiable and, therefore, is treated as a form of missing data. Some elements, such as data collection interval and data collection period, include many attributes identified as unknown. Since all attributes are treated as categorical variables in the model, unknown values can be misinterpreted as informative. To address this issue, we incorporate a penalty into the reward function based on the proportion of unknown values. This adjustment prevents the model from overestimating the informational value of elements with unknown attributes. Based on the state, action, and reward defined in Table 6, the Bellman optimality equation for this problem can be written as Equation (8).

$$Q^{*}(s,a)=r(x)+\gamma \underset{a^{\prime }\in E\setminus (S\cup \{x\})}{\mathit{max}}{Q}^{*}(s\prime ,a\prime )$$

(8)

In this equation, $s^{\prime }$ denotes the next state obtained by adding the selected element $x$ to the set of previously chosen elements (i.e., $s^{\prime }=E(S\cup \{x\})$). This formulation explicitly incorporates the reward function defined in Table 6 and reflects how the choice of element $x$ influences subsequent states in the SDM process.

Additionally, the Q-learning model we developed is trained based on the hyperparameters as shown in

Table 7. Hyperparameters of the Q-learning model.

Learning Rate ($\alpha$)	0.5
Discount Factor ($\gamma$)	0.9
Exploration	$\epsilon$-greedy (initial value: 1.0, decay rate: 0.9, minimum value: 0.5)

. These hyperparameters are set referring to the previous studies [42,43].

The algorithm operates iteratively, as illustrated in Figure 1. The algorithm begins with an initial order list that includes the two predefined elements (purpose and task), forming the initial selection set $S$. The remaining elements in $E\backslash S$ comprise the candidate pool for the next action. At each decision stage, a Q-table is initialized to store the expected Q-value for each selectable element. The agent explores multiple episodes, balancing the exploration of new actions and the exploitation of previously learned Q-values. For each action $x\in E\backslash S$, the reward is computed based on CE reduction. After learning (i.e., exploration), the agent selects the element with the highest Q-value as the optimal action $a^{*}$. This selected element is then added to the order list, and the state is updated by including it in $S$. The process repeats until all elements are selected (i.e., $E\backslash S=\varnothing$). This algorithm yields a sequentially optimized order of decision elements that minimizes CE at each stage while accounting for a data quality issue with unknown attributes through reward penalization with $u_x$.

4. Results and Discussion

This section presents the order of SDM for AI adoption in manufacturing, as derived from the proposed model, and provides simulation-based validation results to evaluate whether the derived order is indeed efficient. Following the predetermined order of purpose and task, the proposed model in this study identified the decision sequence of the remaining six elements that most effectively reduces uncertainty: (i) sensor, (ii) data collection interval, (iii) data dimension, (iv) AI technique, (v) data type, and (vi) data collection period. This order is derived by selecting the element with the highest Q-value at each decision stage. The Q-values calculated for each element at each decision stage are summarized in

Table 8. Stage-wise summary of Q-values for six decision elements.

	Sensor	Data Type	Data Collection Interval	Data Collection Period	Data Dimension	AI Technique
Stage 1	4.770	4.442	3.254	1.297	3.873	3.283
Stage 2		0.383	0.525	0.222	0.274	0.438
Stage 3		0.115		0.096	0.128	0.127
Stage 4		0.031		0.032		0.032
Stage 5		0.000		0.000
Stage 6				0.000

.

To validate whether the model-derived order is an efficient solution to SDM problems, we conduct a simulation in which each trial (hereafter referred to as an ‘episode’) demonstrates a complete sequence of decision-making. Specifically, each episode determines (i) the order of six decision-making elements and (ii) the attribute selected for each element. Once the order and attributes of all six elements are finalized, the episode is considered complete. Additionally, the order and attributes of purpose and task are predefined at the beginning of each episode and are not included in the element ordering and attribute selection.

We define four simulation cases that differ depending on the level of knowledge about the decision-making order of six elements: (a) full order known, (b) relative order known for two elements, (c) exact positions known for two elements, and (d) no order known. These four cases are illustrated in Figure 2.

In Case (a), decisions are made according to the SDM order derived from our CE-based Q-learning model (hereafter, the model-derived order). In contrast, Cases (b)–(d) do not represent additional learned methods or outcomes; they are reference scenarios that partially use the model-derived order or do not use it at all. Case (b) assumes that the relative order between two elements in the model-derived order is known. For instance, information such as ‘sensor must be selected before data type’ or ‘AI technique should be decided before the data collection period’ is assumed to be known. In Case (b), the two elements of the known pairwise order are randomly selected first, and then the remaining four elements are randomly ordered with a uniform distribution. In Case (c), the exact positions of two elements in the model-derived order are known. For example, information such as ‘sensor should be selected first, and data dimension should be placed third in the decision order’ is assumed to be known. Also in Case (c), the two elements of the known positions are randomly selected first, and then the other four elements are randomly ordered with a uniform distribution. In Case (d), the decision-maker is assumed not to know the decision order at all, and thus all elements are randomly ordered with a uniform distribution.

When the order of six elements is determined, an attribute is selected for the element assigned to each stage. The attribute selection follows rule-based constraints defined in

Table 9. List of rule-based constraints applied in the simulation.

No.	If (Trigger Condition)	Then (Constraint)
1	Sensor = Camera	Data type = Image
2	Sensor = Camera	Data dimension $\in$ {<100 × 100 (2D), >100 × 100 (2D)}
3	Sensor = Sound	Data dimension $\in$ {<10 (1D), 10~100 (1D)}
4	Sensor = Sound	Data collection interval = <0.01 s
5	Sensor $\in$ {Energy, Force, Position, Pressure, Rotation, Sound, Velocity/Acceleration, Vibration}	Data type = Numeric
6	Data type = Image	Data dimension $\in$ {<100 × 100 (2D), >100 × 100 (2D)}
7	Data type = Image	Sensor $\in$ {Camera, Others}
8	AI technique ≠ NN	Data dimension $\in$ {<10 (1D), 10~100 (1D)}
9	Data dimension $\in$ {<100 × 100 (2D), >100 × 100 (2D)}	AI technique = NN

to ensure that the simulation reflects realistic conditions. These constraints are based not on specialized domain knowledge, but rather on intuitive and widely accepted interrelationships among elements. For example, there are natural links between certain sensors and data types, or the typical requirements of data dimensions for different AI techniques. More specifically, if a sensor is of a camera type, the resulting data must be in image format, and correspondingly, the data dimension must be 2D. Additionally, except for NN models, most AI techniques cannot effectively capture spatial information in matrix-structured data (i.e., 2D data) unless the data are flattened into 1D form. Therefore, these constraints are essential to prevent the simulation from generating infeasible or unrealistic decision orders that would not be applicable in real-world SDM applications.

Constraints in Table 9 represent directional dependencies between elements. That is, each constraint consists of a trigger condition—the selection of a specific attribute of an element—and a resulting constraint that restricts the feasible attribute of specific elements. During the simulation, attributes are selected randomly from the set of candidates using a uniform distribution by default. However, if a previously selected attribute triggers a constraint, the current element’s attribute is restricted to the feasible attributes allowed by the constraint. In addition, if a constraint allows multiple feasible attributes, one of them is randomly selected using a uniform distribution. For example, if the attribute selected for the data type at the previous stage is ‘image’, then the sensor type allowable at the current stage is restricted to ‘camera’ or ‘others’ due to Constraint 7 in Table 9.

The constraints are not only applied during the attribute selection process but also used as evaluation criteria to assess the feasibility of each simulation case. To this end, we define and detect two types of constraint-related issues during the simulation: constraint violations and constraint conflicts.

A constraint violation occurs when a constraint is triggered but not satisfied due to the decision order. This happens when the element referring to the trigger condition (i.e., ‘if’) intervenes after the element referring to the constraint (i.e., ‘then’). For example, assume that the sensor is selected as ‘camera’ at a later stage, but the data type was already set to ‘numeric’ at an earlier stage. In this case, Constraint 1 is violated. To resolve this violation, the simulation backtracks to the earlier stage and revises the attribute of the already selected element to satisfy the constraint (e.g., ‘numeric’ to ‘image’).

A constraint conflict occurs when two or more constraints require conditions having no intersection to be satisfied. This happens when earlier decisions lead to contradictory requirements. For instance, assume that the AI technique is selected as ‘tree’, which requires a 1D data dimension (Constraint 8), while the data type is set to ‘image’, which requires a 2D data dimension (Constraint 6). These constraints conflict because no single data dimension can satisfy both constraints simultaneously. In this case, the simulation performs a backtracking from the stage determining data dimension to the stage determining data type (the latter-decided element). Then, the attribute of data type is revised (e.g., ‘image’ to ‘numeric’) to resolve the conflict.

The attribute selection and backtracking process is repeated until no further backtracking is required, ensuring that all episodes are completed successfully. Constraint violation and conflict detection are performed at every stage of each episode. If backtracking occurs at any point within an episode, that episode is labeled as a ‘backtracked episode’. The total number of backtracked episodes is used as a key indicator of the robustness and feasibility of the decision order for each simulation case.

For each of the four cases, 1000 episodes are simulated. The number of backtracked episodes for each case is as follows: zero for Case (a), 60 for Case (b), 84 for Case (c), and 81 for Case (d). Since no backtracking occurred in Case (a), the model-derived order can be regarded as the most efficient decision-making order. Figure 3 visualizes one example episode out of 1000 episodes following each case. In particular, examples of backtracked episodes for Cases (b), (c), and (d) are presented in Figure 3; elements with predefined order information are highlighted using underlines for each case.

The Case (a) example shows that no constraint violations or conflicts occurred when decisions are made according to the model-derived order, resulting in only forward decision flow and no backtracking. This result suggests that the model-derived order is appropriately structured so that trigger conditions are always determined before the corresponding constraints are applied. Thus, it prevents infeasible decisions and conflicts among constraints. In contrast, in Case (b), only the information that the AI technique should be decided before the data type is provided. In this example, Constraint 8 conflicts with Constraint 6, since the AI technique selected at Stage 1 is ‘tree’ (i.e., not ‘NN’), which requires the data dimension in Stage 4 to be 1D (Constraint 8). However, at Stage 3, the data type is set to ‘image’, which requires the data dimension to be 2D (Constraint 6). This contradiction leads to a violation of the later-applied Constraint 6. Thus, the process backtracks from Stage 4 to Stage 3 and changes the data type to ‘numeric’. In Case (c), it is assumed that the data collection interval is selected as the second element and the AI technique as the fourth. From the Case (c) example, Constraint 5 is violated at Stage 6; therefore, the decision process rolls back to Stage 1 to revise the attribute of the data type to ‘numeric’. Meanwhile, in Case (d), all of the elements are ordered randomly. The example of Case (d) is similar to that of Case (b); Constraints 6 and 8 conflict since the data type is selected as ‘image’ at Stage 1 and the AI technique is selected as ‘SVM’ at Stage 4. In this episode, since Constraint 6 is applied earlier, the process backtracks from Stage 5 to Stage 4, updating the AI technique to ‘NN’.

We quantify the sum of the uncertainty of all remaining decision elements at each stage based on the adjusted CE defined in Equation (9). This formulation is derived from the reward function used in our RL model and represents a modification of the original CE by considering the informational validity. Specifically, it applies a penalty term $(1-u_e)$, accounting for the proportion of unknown attributes in element $e$.

$$Adjusted CE=\sum _{e\in E\backslash S}H(e|S)\cdot (1-u_e)$$

(9)

Table 10. Adjusted CE by simulation case and decision stage (with AUC).

Stage	Case (a)	Case (b)	Case (c)	Case (d)
0 (Start)	5.550	5.550	5.550	5.550
1	1.591	2.927	3.492	3.429
2	0.973	1.801	2.263	2.122
3	0.827	1.051	1.385	1.311
4	0.128	0.550	0.801	0.754
5	0.074	0.187	0.356	0.351
6	0.000	0.000	0.000	0.000
AUC	6.367	9.291	11.072	10.741

and Figure 4 present the adjusted CE values computed at each stage of the SDM process for the four cases. These values represent the average adjusted CE across 1000 episodes for each case at each stage, thereby capturing the general pattern of uncertainty reduction in each case. Among the cases, Case (a) consistently shows the lowest adjusted CE at every stage, followed by Case (b). This indicates that Case (a) reduces uncertainty most rapidly and efficiently across the sequence. In contrast, Cases (c) and (d) exhibit higher CE values at the early and middle stages, indicating that decisions at several stages failed to consider interdependencies comprehensively. These results align with the number of backtracked episodes; Case (a) shows no backtracked episode, and Case (b) shows fewer backtracked episodes compared to Case (c) and Case (d). In addition, to quantify the overall level of uncertainty present throughout the SDM process, the area under the curve (AUC) of adjusted CE is calculated for each case. A smaller AUC indicates a more efficient order in minimizing cumulative uncertainty. Consistent with the stage-wise CE results, Case (a) shows the smallest AUC (6.367), followed by Case (b) (9.291), while Cases (c) and (d) show substantially higher AUC values (11.072 and 10.741, respectively).

From the case studies provided above, one of the key findings is that an efficient SDM order is an order that systematically reduces interdependency-based uncertainty among decision elements at each stage. Reducing such uncertainty not only facilitates more informed decisions but also minimizes the need for decision revision, leading to a more stable and efficient decision-making process. The reason why a decision order that quickly reduces uncertainty leads to fewer instances of backtracking can be understood by considering the interdependencies among decision-making elements. In SDM settings, selecting one element narrows down the feasible options for the remaining elements. CE quantifies how much uncertainty remains about an element given what has already been selected. When an element with high dependency on others is chosen early (i.e., one that results in a large CE reduction), the remaining space of valid decisions becomes more constrained and easier to scrutinize. This naturally decreases the probability of selecting infeasible attributes at later stages, thereby reducing the possibility of violating constraints. In contrast, if uncertainty remains high at the early stages (i.e., CE is not sufficiently reduced), the SDM process may proceed in a way that superficially captures the logical interdependencies among elements, leading to backtracking and revisions. Therefore, ‘minimizing CE early’ can prevent backtracking and contribute to efficient decision flow.

Another key finding from the simulation-based validation results is the critical importance of understanding relative order relationships between decision-making elements. In the AI adoption process for manufacturing, the selection of one element often constrains or directly influences the feasible attributes of subsequent elements. For example, once a specific sensor type is chosen, it naturally restricts the collectible data types and data dimensions. Therefore, when decision-makers know about the relative order between elements (e.g., sensor before data type), this knowledge is useful to avoid infeasible decisions and reduce backtracking. This explains why Case (b), which involved relative pairwise order knowledge, showed fewer backtracked episodes than Case (c), which involved only fixed position information. The findings suggest that establishing relational knowledge among elements is more crucial than knowing the absolute position of a few individual elements when managing complex SDM tasks.

In addition, the simulation results imply that partial knowledge of the decision order does not always lead to better outcomes. Notably, although all six elements are selected randomly in Case (d), backtracking occurs fewer times than in Case (c). This suggests that incomplete or imprecise guidance may mislead decision-makers to inefficient decisions. Especially in SMEs, decisions are often made based on fragmented knowledge or intuition due to limited expertise. Accordingly, structured decision support, such as the model proposed in this study, can offer more reliable guidance than intuition-driven approaches, particularly in environments with complex interdependencies among decision elements.

CE proves to be an effective metric for identifying dependencies between decision-making elements in SDM problems. CE directly quantifies the remaining uncertainties after selections, allowing the model to prioritize decisions that most significantly reduce overall uncertainty. Therefore, CE is highly useful for understanding and modeling the interdependencies among decision elements, which is critical for constructing a coherent and feasible decision order.

The RL model designed in this study, based on a Q-learning algorithm, effectively discovers an efficient order of decision-making elements that minimizes CE at each stage. By exploring different selection paths and maximizing the reduction in CE, the RL agent successfully captures the underlying dependencies among elements without requiring explicit prior knowledge. In addition, by applying a penalty to elements with a high proportion of unknown attributes, the model prevents overestimation of uncertain information and makes more reliable and informed decisions. This approach enables the RL model to optimize the decision order by maximizing information gain. In addition, by intelligently handling data quality issues with unknown attributes, this approach increases the practical robustness of the decision-making process.

5. Conclusions

The adoption of AI in manufacturing is rapidly accelerating. However, many SMMs lack the domain knowledge necessary to make structured decisions regarding AI adoption. To address this challenge, we propose a novel, data-driven SDM model that supports SMMs in determining an efficient order of decision elements for AI adoption. The model systematically identifies the order that minimizes uncertainty at each decision stage by using CE as a reward function in RL. In addition, the model addresses data quality issues through the application of a Q-learning-based approach.

The development of the model begins with the identification of major decision-making elements and their corresponding attributes for AI adoption in manufacturing, based on a review of 55 prior studies. Then, a dataset of 217 decision cases is constructed to support model training (see Table A1 in Appendix A). Additionally, the model is validated through simulations across different levels of decision-related knowledge. Simulation results demonstrate that following the model-derived order eliminates backtracking during the decision process. Moreover, the model-derived order reduces uncertainty most rapidly and has the lowest cumulative uncertainty across all stages. These results indicate that the proposed model can be effective in generating a practical and feasible decision order. Furthermore, the results highlight that understanding relative pairwise order between elements is more informative and helpful than knowing the exact positions of two elements within the decision-making order. These findings suggest that the proposed model not only optimizes decision flow but also provides a scalable and systematic framework for guiding AI adoption in manufacturing sites. Although this study focuses on data from milling processes, the decision elements and attributes used in our training data can also be commonly applied across other manufacturing processes. Importantly, the proposed SDM model is domain-agnostic and can be applied to a wide range of SDM problems involving interdependent decision elements. Thus, this research contributes not only to manufacturing applications but also to the broader field of decision support by providing a scalable and generalizable methodology for SDM. However, to extend the framework to new domains, issues such as data sparsity and integration with existing decision-support systems may require the model and data to be further refined or expanded.

While the simulation-based validation offers conceptual evidence of the efficacy of the proposed model, it cannot fully represent real industrial SDM processes. Future research should therefore empirically validate the model in the industrial sites to assess both decision quality and practical effectiveness.