Transfer Reinforcement Learning for Combinatorial Optimization Problems

Abstract

Reinforcement learning is an important technique in various fields, particularly in automated machine learning for reinforcement learning (AutoRL). The integration of transfer learning (TL) with AutoRL in combinatorial optimization is an area that requires further research. This paper employs both AutoRL and TL to effectively tackle combinatorial optimization challenges, specifically the asymmetric traveling salesman problem (ATSP) and the sequential ordering problem (SOP). A statistical analysis was conducted to assess the impact of TL on the aforementioned problems. Furthermore, the Auto_TL_RL algorithm was introduced as a novel contribution, combining the AutoRL and TL methodologies. Empirical findings strongly support the effectiveness of this integration, resulting in solutions that were significantly more efficient than conventional techniques, with an 85.7% improvement in the preliminary analysis results. Additionally, the computational time was reduced in 13 instances (i.e., in 92.8% of the simulated problems). The TL-integrated model outperformed the optimal benchmarks, demonstrating its superior convergence. The Auto_TL_RL algorithm design allows for smooth transitions between the ATSP and SOP domains. In a comprehensive evaluation, Auto_TL_RL significantly outperformed traditional methodologies in 78% of the instances analyzed.

1. Introduction

Reinforcement learning (RL) brings together important machine learning (ML) methods [1,2,3,4,5,6,7,8,9]. With RL, an agent learns from interaction with the environment [3,4]. Also, RL is based on Markov decision processes (MDP), and the goal is to maximize the reward received over time [3,4].

A recent field of research studied in association with RL is automated machine learning (AutoML). When AutoML is applied to RL, the approach is called automated reinforcement learning (AutoRL) [10]. AutoRL is an intelligent system designed to select the appropriate conditions for reinforcement learning before learning begins [11]. The use of AutoRL aims to reduce the need for knowledge required by the user as well as reduce the computational cost required during learning [11,12]. These initial conditions can be defined through metalearning, in which the system will use its previous experiences to carry out future activities [12,13,14,15,16].

One way of improving AutoRL approaches is through the use of transfer learning (TL). TL aims to accelerate learning by providing autonomy to the system [17]. The purpose is to transfer knowledge between tasks in different domains and provide various benefits, such as improving the performance of the agent in the target task, improving the agent’s total reward and reducing the time needed to carry out the learning [17,18]. Along these lines, transfer reinforcement learning techniques have been applied in important domains, especially in the areas of robotics [19,20,21,22,23,24] and multiagent systems [25,26,27]. However, the literature has paid little attention to the transfer reinforcement learning for combinatorial optimization problems.

Indeed, the field of combinatorial optimization has several studies using RL [6,28,29,30,31,32,33,34]. Some domains with applications of RL techniques are the traveling salesman problem (TSP) [29,31], k-server problem [32,34], multidimensional knapsack problem [35], vehicle routing [36] and sequential ordering problem (SOP) [37].

The TSP [29,31,33] and the SOP [38,39,40,41] are problems that have great relevance in the literature, which has led to several studies on them. The TSP is a problem with numerous relevant applications, and as a result, several studies have been carried out to solve it [42,43,44,45]. The SOP, like the TSP, also has several important studies, including energy optimization in compilers [46], search optimization [39,47] and parallel machine scaling [48]. Moreover, the TSP and SOP are NP-hard combinatorial optimization problems; that is, in practice, it is necessary to adopt approximate algorithms in the search for better solutions [39]. However, the literature still lacks studies that address tranfer RL in these two problems.

Therefore, the goal of this work is to propose and analyze transfer reinforcement learning between these two relevant combinatorial optimization domains: the ATSP and SOP. For this purpose, instances of the TSPLIB library [49,50,51] and the state–action–reward–state–action (SARSA) algorithm [4] were used. The approach proposed in this work enables generation of a knowledge base in the source domain (ATSP) to transfer learning to the target domain (SOP). In addition to the extensive variety of studies on the ATSP and SOP, these domains were selected due to the similarities in the characteristics of these two combinatorial optimization problems. In this sense, in this paper, three criteria are proposed to be evaluated when carrying out transfer reinforcement learning across combinatorial optimization problems: (1) problems with the same objective function; (2) similar datasets; and (3) transfer from the simpler domain to the more complex domain.

The main contributions include the following:

• A transfer reinforcement learning approach between two classical combinatorial optimization problems. The asymmetric traveling salesman problem is the source domain, and the sequential ordering problem is the objective domain.
• Apply transfer learning to these problems and statistically analyze the results obtained with the transfer.
• Develop a new AutoRL algorithm, apply it to the problems studied and analyze its results.

The present paper is organized into six sections. Section 2 defines the theoretical aspects of the combinatorial optimization problems adopted, RL, AutoML and transfer learning. Section 3 and Section 4 present the methodology and results, respectively. Section 5 shows the comparison of this paper with other literature studies. Finally, Section 6 presents the conclusions of the work.

2. Background

This section will discuss the topics of reinforcement learning, combinatorial optimization, the traveling salesman problem, the SOP, automated machine learning and finally, the transfer learning technique.

2.1. Reinforcement Learning

Reinforcement learning is based on Markov decision processes [3,4]. An MDP is structured into a finite set of states (S), a finite set of actions (A), a finite set of reinforcements (R) and a state transition model (T) [3,4].

In RL, an agent learns through trial and error to make decisions in an environment. Basically, the learning agent performs a sequence of three steps in several repetitions:

(i)

Perceive the current state (s);

(ii)

Perform an action (a);

(iii)

Receive a reward ($r(s,a)$).

In addition, at each instant of time (t), a learning matrix (Q) is updated which stores the knowledge learned. This Q matrix has the dimension given by the number of states in relation to the number of actions in the model.

Equation (1) presents the method for updating the Q matrix using the SARSA algorithm [4]:

$$Q_{t+1}=Q_t(s,a)+\alpha [r(s,a)+\gamma Q_t(s^{\prime },a^{\prime })-Q_t(s,a)]$$

(1)

where s is the state, a the action at time t, $r(s,a)$ is the reward for executing a in s, $s^{\prime }$ is the new state, $a^{\prime }$ is the new action selected, $Q_t$ and $Q_{t+1}$ are matrices at the current time and at $t+1$, respectively, $\alpha$ is the learning rate and $\gamma$ is the discount factor.

Algorithm 1 represents the SARSA algorithm [4].

Algorithm 1: SARSA algorithm
1	Set the parameters: $\alpha ,\gamma \mathrm{e} ϵ$
2	In each s,a do Q(s,a) = 0
3	Observe the state s
4	Select action a using policy $ϵ$-greedy
5	do
6	Run the action a
7	Receive the immediate reward r(s,a)
8	Observe the new state s′
9	Select action a′ using policy $ϵ$-greedy
10	$Q(s_t,a_t)=Q(s_t,a_t)+\alpha [r(s_t,a_t)+\gamma Q(s^{\prime },a^{\prime })-Q(s_t,a_t)]$
11	s = s′
12	a = a′
13	while the stop criterion is satisfied;

In Algorithm 1, the $ϵ$-$greedy$ [4] action selection policy is adopted. This method uses the $ϵ$ parameter to control gluttony and randomness in decision making. For example, if $ϵ=0.1$, then the system will select random actions in $10\%$ of the cases and the best-estimated actions for each state in the learning matrix in $90\%$ of the situations.

2.2. Combinatorial Optimization

Combinatorial optimization problems aim to maximize or minimize an objective function defined in a certain finite domain that classifies the solution to the problem as optimal [52,53]. Briefly, these problems can be characterized as follows [53,54,55]:

• Decision variables are the criteria that will be manipulated in the search for the optimal solution.
• The objective function is the function that contains the decision variables that will be altered during the search for the best solution to the problem. It should be noted that every combinatorial optimization issue has at least one objective function.
• Restrictions are conditions imposed on the problem to ensure that the solution found is feasible. It is important to note that it is not mandatory for a problem to have constraints, in which case all solutions are considered feasible.

The literature contains various optimization problems, including the following [52,55,56]:

• The traveling salesman problem;
• The knapsack problem;
• The sequential ordering problem;
• The quadratic assignment problem.

2.3. Traveling Salesman Problem

The traveling salesman problem is one of the best-known combinatorial optimization problems [57]. This method aims to determine a Hamiltonian cycle that passes through all the vertices of a graph. In addition, the problem also aims for the Hamiltonian cycle found to be the one with the lowest cost [52,57,58].

The TSP is made up of a set of cities, where the cashier must visit all the cities in the set, and at the end of the journey, the cashier must return to the initial city. As a restriction, each city in the set must be visited only once, except for the final city, which must be the same as the starting city [57,58].

The TSP can be classified in various ways, including the symmetrical TSP and asymmetrical TSP. In the symmetrical TSP, the cost of moving from node A to node B is the same as the cost of moving from node B to node A. In the case of the asymmetric TSP, the cost varies according to the direction of travel adopted [57]. Thus, the symmetric TSP can be represented by a graph with bidirectional edges, while the asymmetric TSP can be represented by a graph with directional edges [57].

The traveling salesman problem has several applications, and the authors of [52,58] presented some of them:

• Task sequencing;
• Drilling printed circuit boards;
• Analysis of crystal structures;
• Handling of stock items;
• Optimizing the movement of cutting tools;
• Postal delivery routing.

Formulation

There are several possible formulations for the TSP, some of which are presented in [52,59]. The following formulation was presented in [59] in Equations (2)–(6):

$$Minimize\sum _{i=1}^N\sum _{j=1}^N{c}_{ij}{x}_{ij}$$

(2)

subject to

$$\sum _{i=1}^N{x}_{ij}=1(\forall j=1,\dots ,N)$$

(3)

$$\sum _{j=1}^N{x}_{ij}=1(\forall i=1,\dots ,N)$$

(4)

$$x_{ij}\in \{0,1\}(\forall i,j=1,\dots ,N)$$

(5)

$$X=x_{ij}\in S(\forall i,j=1,\dots ,N)$$

(6)

The objective function of the TSP is represented by Equation (2), in which $c_{ij}$ represents the cost between cities i and j, and $x_{ij}$ represents whether the arc (i, j) is part of the solution to the problem. If $x_{ij}$ is equal to one, then the arc is part of the solution. If $x_{ij}$ equals zero, then the arc is not part of the solution. In addition, Equations (3) and (4) ensure that each city will only be visited once. Equation (5), on the other hand, represents the restriction that the value of $x_{ij}$ will always be binary. Finally, Equation (6) guarantees that no sub-routes will be formed when solving the problem [58,59].

2.4. Sequential Ordering Problem

The sequential ordering problem is an NP-hard combinatorial optimization problem and a variation of the asymmetric traveling salesman problem (ATSP), with the addition of precedence constraints [39,47]. Thus, as with the asymmetric TSP, the SOP also aims to find a Hamiltonian cycle of a minimum cost in a directed graph [39]. Thus, the search for the optimal solution to this problem becomes impractical, resulting in the need to apply algorithms to find the best approximate solutions [39].

Consider that Figure 1 is a representation of an SOP problem. In this case, Figure 1a shows the graph that depicts this problem. Figure 1b shows the constraints of this problem. Assuming that the agent starts its trajectory at vertex A, according to the problem constraint, some valid routes are $A\to B\to C\to D\to E\to A$ or $A\to E\to B\to C\to D\to A$. On the other hand, routes $A\to C\to D\to B\to E\to A$ and $A\to D\to E\to B\to C\to A$ are some invalid routes, as they do not respect the problem’s precedence restriction.

In the SOP, $c_{ij}$ also represents the cost of the arc. When $c_{ij}\ge 0$, this value will represent the cost of moving from city i to city j. When $c_{ij}=-1$, this value indicates that there is a precedence constraint. In this case, city j must precede city i [39].

In [37], the authors presented a formulation for the SOP based on the TSP formulation in [59]. This adds the following restriction to the TSP formulation presented previously, according to Equation (7):

$$c_{ij}\ge 0\vee c_{ij}=-1\wedge c_{ji}\ge 0(\forall i,j=1,\cdots ,N)$$

(7)

This new constraint aims to satisfy the existing order of precedence restriction in the SOP.

2.5. Automated Machine Learning

Automated machine learning was developed with the main aim of reducing the human effort required when adjusting the learning settings [60,61]. Thus, AutoML is a technique used to define the parameters and algorithms that will be adopted during learning in order to obtain better results, given that the settings will be made according to the problem being studied [62,63].

In this context, due to the improvements observed through the use of AutoML and the growing need for increasingly robust learning systems, in order to cope with the abundance of data that is constantly emerging, this technique has also been used to automate other stages of the learning process [60,63,64].

Metalearning is one of the topics related to AutoML and refers to the system’s ability to learn how to learn [12,65,66]. In this sense, the goal of metalearning is to reuse previous experiences in tasks that will be performed in the future [12,13,65]. In this way, the system does not have to learn from scratch and can adapt to the current situation based on its previous experiences [67]. Therefore, there will be a reduction in the effort used to carry out the next tasks compared with the effort used to carry out the previous tasks [12,68]. It is worth noting that the greater the similarity between the tasks (the tasks already carried out and the tasks to be carried out), the greater the chances of obtaining good results by reusing knowledge obtained in previous tasks [14,68].

2.6. Transfer Learning

The main idea of transfer learning techniques is to use the knowledge that has already been acquired (tasks that have already been carried out) in related problems [17,26,69]. In this sense, some objectives of TL methods are to improve performance and reduce the time needed to learn a complex task [17].

For transfer learning, the knowledge base is first generated in a source domain and then applied to the target domain [17,18]. This approach is extremely beneficial in various situations, particularly when training for the target problem is complex [70]. By conducting experiments on a source problem, knowledge can be transferred to the target problem.

There are several possible applications for TL, some of which involve the following:

• Image analysis [71,72,73];
• Pattern recognition [74];
• Optimization problems [75].

One method for transfer reinforcement learning involves directly transferring the learning matrix Q as described in [18]. This method adopts the Q values from the source task as the starting point for the learning matrix in the objective domain. Equation (8) represents the direct transfer of RL between domains [18]:

$$\forall s,\forall a,Q_{objective}(s,a)=Q_{source}(s,a)$$

(8)

3. Methodology

The transfer learning approach proposed in this paper aims to improve the performance of RL algorithms for solving combinatorial optimization problems. For this, concepts from the meta-learning research field were adopted [12]. Initially, the application of representational transfer stands out, which refers to the training of source and target models at different times; in other words, metaknowledge is transferred based on previous experience acquired. The second concept is the search for similarities in characteristics between datasets in order to improve the performance of learning algorithms. Thus, transfer of knowledge across tasks is an area of meta-learning that seeks to explore the experience obtained in previously learned datasets, domes or problems.

Following this line, this paper proposes three topics to be evaluated when performing transfer RL between combinatorial optimization problems:

• Problems with the same objective function: The objective function is a main characteristic of the mathematical modeling of a combinatorial optimization problem. This was a relevant criterion in deciding the two problems evaluated in this paper (the ATSP and SOP), considering that both aim to minimize the distance of a route.
• Similar datasets: The similarity between datasets can be assessed through the analysis of metafeatures, such as by performing descriptive statistics [12]. In this paper, the simulated domains have instances in the TSPLIB library. Furthermore, the instances selected for the target problem (SOP) originate from the source domain (ATSP), adding some variations such as precedence restrictions. These characteristics reinforce the similarities across the evaluated datasets.
• Transfer from the simpler domain to the more complex domain: This point reflects the relevance of decreasing the computational cost in the objective problem while promoting accelerated learning and advancing performance in the best solution. In this paper, the ATSP is the source domain (simplest), and the SOP is the objective domain (more complex), as the second problem adds precedence restrictions.

In sequence, this section describes in detail the methodology adopted to carry out this work. Initially, the dataset was built from the library TSPLIB (http://comopt.ifi.uni-heidelberg.de/software/TSPLIB95/, accessed on 1 December 2023). In this context, the specific problems to be investigated and their respective instances were selected. A model based on reinforcement learning was then developed, using the SARSA algorithm to tackle combinatorial optimization problems. As for the application of the transfer learning technique, the strategy was outlined in three phases: the construction of the knowledge base, the execution of the necessary experiments and the formulation of the methodology for analysis. Finally, an automated reinforcement transfer learning algorithm was created and structured in two stages: proposing an algorithm called Auto_TL_RL and carrying out practical experiments to validate and evaluate the algorithm.

3.1. Dataset

The TSPLIB [49] library is a data repository with instances for case studies of combinatorial optimization problems. TSPLIB is frequently adopted in the literature [30,31,37,39,41], and therefore, it was selected for this paper. Among the data available in the repository are data from problems such as the following [50]:

• The symmetric traveling salesman problem: the cost of traveling between two nodes is the same, regardless of the direction of travel;
• The asymmetric traveling salesman problem: the cost of traveling between two nodes depends on the direction of travel;
• The sequential ordering problem: this problem has precedence restrictions and also considers that the cost of traveling between two nodes depends on the direction adopted for travel.

The instances selected for use in the experiments are shown in

Table 1. ATSP instances.

Problem	Nodes	Best Known Solution
br17	17	39
p43	43	5620
ry48p	48	14,422
ft53	53	6905
ft70	70	38,673
kro124p	100	36,230

(ATSP) and

Table 2. SOP instances.

Problem	Nodes	Best Known Solution
br17.10	18	55
br17.12	18	55
p43.1	44	28,140
p43.2	44	28,480
p43.3	44	28,835
p43.4	44	83,005
ry48p.1	49	14,422
ry48p.2	49	16,074
ry48p.3	49	19,490
ry48p.4	49	31,446
ft53.1	54	7531
ft53.2	54	8026
ft53.3	54	10,262
ft53.4	54	14,425
ft70.1	71	39,313
ft70.2	71	40,101
ft70.3	71	42,535
ft70.4	71	53,530
kro124p.1	101	38,762
kro124p.2	101	39,841
kro124p.3	101	43,904
kro124p.4	101	73,021

(SOP). These tables have three columns: problem, nodes and optimal:

• Problem: instance name;
• Nodes: number of nodes in the problem;
• Best known solution: the best known value presented by TSPLIB.

To exemplify the data on the adopted instances,

Table 3. Example of ATSP instance br17 (TSPLIB).

9999	3	5	48	48	8	8	5	5	3	3	0	3	5	8	8	5
3	9999	3	48	48	8	8	5	5	0	0	3	0	3	8	8	5
5	3	9999	72	72	48	48	24	24	3	3	5	3	0	48	48	24
48	48	74	9999	0	6	6	12	12	48	48	48	48	74	6	6	12
48	48	74	0	9999	6	6	12	12	48	48	48	48	74	6	6	12
8	8	50	6	6	9999	0	8	8	8	8	8	8	50	0	0	8
8	8	50	6	6	0	9999	8	8	8	8	8	8	50	0	0	8
5	5	26	12	12	8	8	9999	0	5	5	5	5	26	8	8	0
5	5	26	12	12	8	8	0	9999	5	5	5	5	26	8	8	0
3	0	3	48	48	8	8	5	5	9999	0	3	0	3	8	8	5
3	0	3	48	48	8	8	5	5	0	9999	3	0	3	8	8	5
0	3	5	48	48	8	8	5	5	3	3	9999	3	5	8	8	5
3	0	3	48	48	8	8	5	5	0	0	3	9999	3	8	8	5
5	3	0	72	72	48	48	24	24	3	3	5	3	9999	48	48	24
8	8	50	6	6	0	0	8	8	8	8	8	8	50	9999	0	8
8	8	50	6	6	0	0	8	8	8	8	8	8	50	0	9999	8
5	5	26	12	12	8	8	0	0	5	5	5	5	26	8	8	9999

and

Table 4. Example of SOP instance br17.12 (TSPLIB).

0	3	5	48	48	8	8	5	5	3	3	0	3	5	8	8	5	1,000,000
$-1$	0	3	48	$-1$	$-1$	8	5	$-1$	0	0	3	0	3	8	$-1$	5	3
$-1$	3	0	72	$-1$	48	48	24	$-1$	$-1$	3	5	3	0	48	$-1$	24	5
$-1$	48	74	0	0	6	6	12	$-1$	48	48	48	48	74	6	6	12	48
$-1$	48	74	0	0	6	6	12	$-1$	48	48	48	48	74	6	6	12	48
$-1$	8	50	6	6	0	0	8	8	8	8	8	8	50	0	0	8	8
$-1$	8	50	6	6	0	0	8	8	8	8	8	8	50	0	0	8	8
$-1$	5	26	12	12	$-1$	8	0	0	5	5	5	$-1$	26	8	8	0	5
$-1$	5	26	12	12	8	8	0	0	5	5	5	5	26	8	8	0	5
$-1$	0	3	48	$-1$	8	8	5	$-1$	0	0	3	0	3	8	8	5	3
-1	0	3	48	48	8	8	5	5	0	0	3	0	3	8	8	5	3
$-1$	3	5	48	48	8	8	5	5	3	3	0	3	5	8	8	5	0
$-1$	0	3	48	48	$-1$	8	5	5	0	0	3	0	3	8	8	5	3
$-1$	3	0	72	$-1$	48	48	24	$-1$	3	3	5	3	0	48	48	24	5
$-1$	8	50	6	6	$-1$	0	$-1$	8	8	8	8	$-1$	50	0	0	8	8
$-1$	8	50	6	$-1$	0	0	8	$-1$	8	8	8	8	50	0	0	8	8
$-1$	5	26	12	12	8	8	0	0	5	5	5	5	26	8	8	0	5
$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	$-1$	0

present instances br17 (ATSP) and br17.12 (SOP), respectively.

Table 3 and Table 4 reveal some characteristics of the data matrices adopted in this paper. Initially, it is worth highlighting that the values in the matrix cells represent the weights between vertices. For example, looking at the first row and second column of Table 3, $c_{ij}$ is equal to three, as it represents the weight of traveling from $i=1$ to $j=2$. It was also observed that the dimension of the matrix revealed the number of nodes in the instance. In this sense, br17 had a dimension of 17 (Table 3), while br17.12 had a dimension of 18 (Table 4). This relationship shows that the SOP instances had dimensions of $N+1$ in relation to the corresponding ATSP instances, where N is the dimension of the ATSP instance. Note also the changes made to Table 4 in relation to Table 4, where some data positions were replaced with the value “$-1$”, indicating the addition of a precedence constraint. For example, in the second row and fifth column in Table 3, the value is $c_{ij}=48$. On the other hand, in Table 4, this value was changed to “$-1$”, becoming an SOP precedence restriction.

3.2. Reinforcement Learning Model

The RL system developed was designed to apply the SARSA algorithm to experiments with combinatorial optimization problems: the ATSP and SOP. For this purpose, the model structure (actions, states and reinforcements) was adopted based on works in the literature [29,30,31,37,76]:

• States are the locations that must be visited to form the route. Thus, the number of states varies according to the number of nodes (N) in the instance.
• Actions represent the possible movements between locations (states). The initial number of actions is equivalent to the number of states in the model. However, the actions available for execution vary according to the cities already visited when developing the route.
• Reinforcements are the cost of travel ($c_{ij}$) between the departure city (i) and the destination city (j), given as a function of the cost of travel ($c_{ij}$). The greater the distance between the nodes, the more negative the reinforcement will be, according to Equation (9):

  $$R=-c_{ij}$$

  (9)

Also, the RLSOP algorithm [37] was adopted to deal with the precedence restrictions of the SOP. The RLSOP (Algorithm 2) checks whether the actions (arrival location) selected by the $ϵ$-$greedy$ method have precedence restrictions. If this is the case, then another destination is selected and checked again.

Algorithm 2: Algorithm for analyzing precedence restrictions in the selection of actions from RL to SOP (RLSOP) [37]
1	a_t = e-greedy();
2	cont = 0;
3	while cont == 0 do
4	if there are precedence restrictions for the selected action then
5	if at least one action corresponding to the precedence constraints of a_t has not yet been selected then
6	cont = 0;
7	else
8	cont = 1;
9	end
10	end
11	if cont == 0 then
12	remove the action a_t from the list of available actions at time t;
13	a_t = e-greedy();
14	end
15	end
16	Return to_t;

3.3. Transfer Learning Approach

To implement the transfer learning technique, it is necessary to follow three steps:

• Generation of the knowledge base;
• Experiments for transfer learning;
• Elaboration of the methodology for analysis.

3.3.1. Generation of the Knowledge Base

The knowledge base generation stage was conducted through experiments with TSPLIB instances of the asymmetric traveling salesman problem. Among the instances shown in Table 1, the following ATSP instances were selected for use in this approach: br17, p43, ft53 and kro124p. SOP instances derived from the selected ATSP instances were also used (Table 2).

The experiments with each of the four ATSP instances were configured with 10,000 episodes, where one episode is equivalent to a complete route between the nodes. The learning matrix in these simulations was initialized with zeros. In addition, the parameters were set to $\alpha =0.75$, $\gamma =0.15$ and $ϵ=0.01$, based on the results in [37,76].

At the end of each simulation, the final learning matrix (QATSP) was stored. In this sense, four QATSP matrices were generated, with one per ATSP instance (br17, p43, ft53 and kro124p). This knowledge base was adopted in the transfer learning experiments for the SOP domain, which are explained in the next subsection.

3.3.2. Experiments for Transfer Learning

In this phase, experiments were carried out to assess the influence of knowledge transfer between the ATSP (source) and SOP (goal) domains.

Table 5. SOP problems adopted and respective numbers of nodes, precedence constraints and best known solution values according to TSPLIB. The number of constraints refers to the number of values of $c_{ji}=-1$ in the instance.

Problem	Nodes	Restrictions	Best Known Solution
br17.10	18	48	55
br17.12	18	55	55
ft53.1	54	117	7536
ft53.2	54	135	8026
ft53.3	54	322	10,262
ft53.4	54	865	14,425
kro124p.1	101	232	38,762
kro124p.2	101	267	39,841
kro124p.3	101	465	43,904
kro124p.4	101	2504	73,021
p43.1	44	96	28,140
p43.2	44	119	28,480
p43.3	44	181	28,835
p43.4	44	581	83,005

shows the 14 SOP instances adopted in this stage, as well as their respective optimal values, number of nodes and precedence restrictions.

Experiments were carried out using two initial conditions for the learning matrix (Q):

• Q0, meaning without transfer learning. The learning matrix was initialized with all null values.
• The QATSP, adopting the knowledge base generated from the experiments with the source domain (ATSP).

In order to use the knowledge base (QATSP), adjustments had to be made to the original learning matrices (experiments with the ATSP). The reason for this is that the state spaces of the ATSP and SOP domains are different. For example, the kro124p instance (ATSP source) has 100 nodes, while the equivalent SOP problems contain $N+1$ locations: kro124.p1, kro124.p2, kro124.p3 and kro124.p4. Thus, a row and a column (with zeros) were added to the QATSP knowledge base matrices for use by the SOP domain.

Figure 2 illustrates how the transfer RL process takes place. Normally, during experiments, the learning matrix is initialized with zeros (Figure 2b). Assuming that Figure 2a shows the learning matrix of an ATSP instance at the end of the experiment, with the proposed learning transfer approach, the experiment learning matrix of an SOP instance would initially be filled in as shown in Figure 2c.

The experiments with each of the SOP problems and initial conditions (Q0 and QATSP) were run with 100 episodes in 10 epochs (repetitions). It is worth noting that only 100 episodes were used in this stage, as the goal was to analyze the adoption of transfer learning as a method to accelerate RL in the objective domain [17]. In this way, the aim was to assess whether the adoption of the knowledge base (QATSP) reproduced good results in a few episodes of SOP simulation.

The parameters for this stage were defined in the same way as in the previous section: $\alpha =0.75$, $\gamma =0.15$ and $ϵ=0.01$.

3.3.3. Analysis Methodology

The analysis methodology proposed in this paper has three stages. This methodology was based on metrics found in the literature [17,26]:

(i)

Preliminary analysis;

(ii)

Computational time analysis;

(iii)

Results visualization and interpretation.

The preliminary analysis compares the average results achieved in the SOP instances with and without transfer learning from the source domain (ATSP). This metric is similar to the “total reward” [17], which adopts the total reinforcement received by the agent during learning.

The second stage aims to evaluate the differences between the computational times of the simulations with Q0 and the QATSP. According to [17], one possible goal of transferring knowledge between domains is to reduce the time it takes to learn a complex task.

In these first two steps, the t test is used to compare the means of two independent samples. This statistical method assesses whether two population means (${\mu }_1$ and ${\mu }_2$) are statistically equal or different [77] through the following hypotheses:

$$\left\{\begin{array}{cc}\hfill H_0:& {\mu }_1={\mu }_2,\\ \hfill H_1:& {\mu }_1\ne {\mu }_2.\end{array}\right.$$

Adopting a significance level of 5%, the decision rule is as follows. If $p>0.05$, then the means are considered to be statistically equal. On the other hand, when $p\le 0.05$, the means are concluded to be statistically different. To validate the results, it is necessary to test the normality assumption for each of the independent samples. In this study, the Kolmogorov–Smirnov (KS) test was adopted [78], where the normality assumption was satisfied as long as the p value of this test was greater than 5%. Thus, after verifying that the assumptions were guaranteed by the KS test for all the samples, the t test was applied to compare the average results for the distance on the route (preliminary analysis) and computational time for each of the 14 simulated instances.

Finally, results visualization and interpretation evaluates the learning curves of the following SOP instances: krop124p.1, krop124p.2, krop124p.3 and krop124p.4. Three metrics were analyzed: the distance calculated in the first episode ($d_0$), the distance in the last episode ($d_f$) and the smallest solution found ($d_{min}$). Evaluating the solution of the first (“jump start”) and last (“asymptotic performance”) episodes was important for seeing the difference in the results of Q0 and the QATSP in these situations [17]. The value of $d_{min}$, on the other hand, allowed us to compare whether adopting a knowledge base (QATSP) led to better solutions.

3.4. Automated Transfer Reinforcement Learning Method

This section introduces the automated reinforcement learning transfer method, which aims to automate the transfer of reinforcement learning between the ATSP and SOP problems. The method consists of two stages:

• A new automated transfer reinforcement learning algorithm (Auto_TL_RL).
• Automated transfer reinforcement learning experiments.

3.4.1. Automated Transfer Reinforcement Learning Algorithm (Auto_TL_RL)

The flowchart depicted in Figure 3 illustrates the proposed automated transfer reinforcement learning algorithm flow, which has been named Auto_TL_RL. Initially, an SOP instance is selected, and if a knowledge base already exists for the selected SOP instance, then the problem is executed directly. Otherwise, the knowledge base is generated from an ATSP instance, this database is then stored, and finally the SOP is executed.

Algorithm 3 represents the proposed Auto_TL_RL algorithm procedurally, which was developed using the R language. It is worth noting that the proposed algorithm uses the SARSA algorithm to perform the ATSP and SOP.

Algorithm 3: Auto_TL_RL Algorithm
1	Specify the SOP instance to be executed
2	Extract the size of the SOP instance
3	Set the parameter: $ϵ$
4	Set the number of episodes: 1000
	Stage 1
5	Checking the existence of the knowledge base
	Stage 2
6	Specify the ATSP instance that will be executed
7	Extract the size of the ATSP instance
8	Set the parameters: $\alpha$ and $\gamma$
9	Set the number of epochs: 5
10	foreach ${\alpha }_t$ ∈ $\alpha$ do
11	foreach ${\gamma }_t$ ∈ $\gamma$ do
12	for epoch to numberEpochs do
13	SARSA($\alpha$, $\gamma$, sizeOfInstance)
14	end
15	end
16	end
17	Stores the database generated
	Stage 3
18	Set the parameters: $\alpha$ and $\gamma$
19	Set the number of epochs: 10
20	for epoch to numberEpochs do
21	SARSA($\alpha$, $\gamma$, sizeOfInstance)
22	end

Initially, the user must select the instance of the SOP to execute (line 1), and from this, the size of the instance is extracted (line 2). Next, the value of the $ϵ$-greedy parameter and the number of episodes that will be executed during learning must be defined (lines 3 and 4). Next, the execution of the algorithm is divided into three stages. The first stage checks for the existence of the database related to the selected problem (line 5). To check for the existence of the knowledge base, a document is used that records the name of the instance, the dimension, the status of the knowledge base (whether it exists or not) and the name of the file that stores the knowledge base corresponding to the instance analyzed.

The second path is taken if there is no pre-existing knowledge base for the selected SOP instance. It defines an ATSP instance and extracts its dimension (lines 6 and 7). Furthermore, the values for the learning rate, the discount factor and the number of epochs are defined (lines 8 and 9). In this sense, with the learning rate and discount factor values defined, different parameter combinations are performed. For each simulated parameter combination, the selected ATSP instance is executed for the defined number of epochs (lines 10–16). At the end of the execution, the knowledge base that resulted in the shortest route distance in the ATSP instance used is stored among all generated databases (line 17), allowing it to be reused in future applications. Finally, in the third step, the SOP is executed (lines 18–22) using the knowledge base generated in the previous step.

The second path is adopted when there is no pre-existing knowledge base for the selected SOP instance. Thus, an ATSP instance is defined, and its dimension is extracted (lines 6 and 7). Furthermore, the values for the learning rate, discount factor and number of epochs are defined (lines 8 and 9). In this sense, with the learning rate and discount factor values established, various combinations of parameters are performed. For each simulated parameter combination, the selected ATSP instance is executed for the defined number of epochs (lines 10–16). At the end of the execution, the knowledge base that resulted in the shortest route distance in the ATSP instance used is stored among all generated databases (line 17), allowing it to be reused in future applications. Finally, the third stage is where the SOP is executed (lines 18–22) using the knowledge base generated in the previous step.

The Auto_TL_RL algorithm in R language has been made available in an open repository to provide detailed visualization of the technical aspects of the code and to ensure its reproducibility (https://github.com/KellyBarbosa/auto_tl_rl, accessed on 12 February 2024).

3.4.2. Automated Transfer Reinforcement Learning Experiments

Automated transfer reinforcement learning experiments were also performed using SOP and ATSP instances provided by the TSPLIB library. The selected ATSP and SOP instances are shown in Table 1 and Table 2, respectively. In this approach, the following ATSP instances were selected: br17, p43, ry48p, ft53 and ft70. In addition to the ATSP problems, the SOP instances that were derived from the selected ATSP instances were also selected.

The parameters, number of epochs and number of episodes to be used in the proposed Auto_TL_RL algorithm (Algorithm 3) during the experiments were also defined. The configurations for the second stage were the same as those presented in [79], and they can be seen in

Table 6. Configurations used during the second stage of the experiment.

Parameters	Quantity	Values
$\alpha$	8	0.01; 0.15; 0.30; 0.45; 0.60; 0.75; 0.90; 0.99
$\gamma$	8	0.01; 0.15; 0.30; 0.45; 0.60; 0.75; 0.90; 0.99
$ϵ$	1	0.01
Combinations	$8\times 8\times 1=64$	-
Epochs per Combination	5	-
Episodes per Epoch	1000	-
Episodes per Combination	$5\times 1000=5000$	-
Total Epochs	$5\times 64=320$	-
Total Episodes	$1000\times 320$ = 320,000	-

. The learning conditions for the final stage followed the same structure. However, some adaptations were made to better suit the experimental scenarios, as shown in

Table 7. Configurations used during the third stage of the experiment.

Parameters	Quantity	Values
$\alpha$	1	0.75
$\gamma$	1	0.15
$ϵ$	1	0.01
Combinations	$1\times 1\times 1=1$	-
Epochs per Combination	10	-
Episodes per Epoch	1000	-
Episodes per Combination	$10\times 1000$ = 10,000	-
Total Epochs	10	-
Total Episodes	$10\times 1000$ = 10,000	-

.

4. Results

This section presents the results of the preliminary analysis and the automated transfer learning experiments.

4.1. Results of Preliminary Analysis

For a more comprehensive understanding, the results of the preliminary analysis are presented and detailed in three separate sections: Preliminary Analysis, Computational Time Analysis, Results Visualization and Interpretation.

4.1.1. Preliminary Analysis

The preliminary analysis compares the solutions (distance on the route) according to the initial learning condition adopted (Q0 or the QATSP). For each instance, the average solution of the 10 simulated repetitions was calculated.

Table 8. Average solution (distance) over the course of learning to solve the SOP, percentage difference (D) between the results of Q0 and the QATSP and results of the t test. There was a significant difference between the mean solutions of Q0 and the QATSP if $p\le 0.05$.

Problem	Q0	QATSP	D(%)	t	p
br17.10	99.6	117.2	17.67	−3.20	0.01
br17.12	92.7	100.8	8.74	−2.14	0.06
ft53.1	19,054.3	10,003.9	−47.50	126.10	0.00
ft53.2	19,735.9	12,057.4	−38.91	138.95	0.00
ft53.3	19,583.6	16,173.2	$-1$7.41	25.40	0.00
ft53.4	19,360.0	18,245.4	−5.76	26.56	0.00
kro124p.1	179,266.4	56,146.9	−68.68	789.13	0.00
kro124p.2	179,859.2	59,280.3	−67.04	432.24	0.00
kro124p.3	168,223.3	71,605.7	−57.43	221.92	0.00
kro124p.4	124,900.0	99,131.0	−20.63	67.09	0.00
p43.1	72,411.4	30,453.1	−57.94	117.94	0.00
p43.2	71,953.6	32,726.0	−54.52	101.90	0.00
p43.3	67,146.8	33,151.5	−50.63	84.45	0.00
p43.4	93,488.1	86,304.4	−7.68	25.96	0.00

shows the results of this stage.

Table 8 shows that the more negative the percentage difference between the results, the greater the efficiency of adopting the knowledge base (QATSP) compared with Q0. It can be seen that the experiments with the QATSP had a lower solution (shorter path) in 12 SOP problems ($85.7\%$) out of 14 instances in total.

Regarding the results of the t test, in 13 SOP instances, there was a significant difference ($p\le 0.05$) between the average route distances. More specifically, of the 13 instances where there were differences, 12 of them showed better results with the use of the knowledge base (QATSP). Only one instance, br17.10, showed a disadvantage when adopting transfer learning.

4.1.2. Computational Time Analysis

Table 9. Average computational times (in seconds) for solving the SOP, percentage difference (D) between the results of Q0 and the QATSP and the results of the t test. There was a significant difference between the mean computational times for Q0 and the QATSP if $p\le 0.05$.

Problem	Q0	QATSP	D(%)	t	p
br17.10	0.33	0.22	−33.33	3.24	0.01
br17.12	0.30	0.22	−26.67	9.62	0.00
ft53.1	0.65	0.59	−9.23	2.87	0.01
ft53.2	0.84	0.66	−21.43	9.95	0.00
ft53.3	1.79	0.89	−50.28	60.65	0.00
ft53.4	2.62	1.27	−51.53	73.37	0.00
kro124p.1	1.63	1.15	−29.45	10.42	0.00
kro124p.2	2.12	1.12	−47.17	73.63	0.00
kro124p.3	3.36	1.57	−53.27	57.10	0.00
kro124p.4	7.61	2.88	−62.16	158.77	0.00
p43.1	0.55	0.59	7.27	$-1$.49	0.15
p43.2	0.79	0.72	−8.86	4.53	0.00
p43.3	1.13	1.08	−4.42	2.73	0.01
p43.4	1.84	1.51	$-1$7.93	27.04	0.00

shows the average computational times of the simulations of the SOP problems according to the initial learning matrix (Q0 or the QATSP), the respective percentage difference and the results of the t test. Again, as described in Table 8, the more negative the percentage value of the difference between the simulation computational times, the more efficient the adoption of the knowledge base (QATSP) with respect to Q0 was.

According to the t test, in 13 instances (of the 14 analyzed)—that is, in $92.8\%$ of the simulated problems—the average execution time was lower when adopting the knowledge base ($p\le 0.05$).

4.1.3. Results Visualization and Interpretation

In this stage, the learning evolution graphs (distance calculated over the episodes) are analyzed for the following SOP problems: kro124p.1, kro124p.2, kro124p.3 and kro124p.4. These four problems are based on kro124p (ATSP), and the data between these instances vary depending on the number and arrangement of precedence constraints between the nodes (see Table 1 and Table 5).

Figure 4, Figure 5, Figure 6 and Figure 7 show that the QATSP learning curves started the episodes closer to the optimal value of the instance than the simulations with Q0. This is most evident when comparing the $d_0$ values. In all the graphs shown, the difference between the solution in the initial episode of the QATSP and Q0 was greater than $0.8\times {10}^5$ (80,000) distance units. This highlights the importance of transferring knowledge between the ATSP and SOP domains in the initial episodes, favoring the acceleration of learning.

In addition, the graphs in Figure 4, Figure 5 and Figure 6 also show the differences between the $d_f$ values for Q0 and the QATSP. For instances kro124p.1, kro124p.2 and kro124p.3, adopting transfer learning resulted in better results in the last episode of the simulation. In the worst case (experiments on problem kro124p.4), the Q0 learning curve still needed around 50 episodes to achieve results close to those presented by the simulations using the ATSP knowledge base.

Finally, it is worth highlighting that for the graphs presented in Figure 4, Figure 5, Figure 6 and Figure 7, employing transfer learning between the ATSP and SOP domains resulted in calculating better solutions during learning. For example, for the kro124p.1 instance, $d_{min}$ = 52,029 when adopting the QATSP, while for learning without prior knowledge, the result was $d_{min}$ = 136,261.

4.2. Results of Automated Transfer Learning

First, for comparison purposes, the selected SOP instances were run without using transfer learning. Next, the knowledge bases were generated using the ATSP instances at the time of execution of problems br17.10, p43.1, ry48p.1, ft53.1 and ft70.1. Thus, at the time of execution of these instances, all the stages of the proposed algorithm were executed.

For the remaining instances, as there was already a pre-existing knowledge base, only the first and third stages of the algorithm were run.

Table 10. Results of running SOP instances without the Auto_TL_RL algorithm and with the Auto_TL_RL algorithm, with the asymmetric traveling salesman problem (ATSP); sequential ordering problem (SOP); best known solution value presented by TSPLIB; and the proposed algorithm (Auto_TL_RL).

ATSP	SOP	Best Known Solution	Without Auto_TL_RL	With Auto_TL_RL
br17	br17.10	55	57	55
	br17.12	55	57	57
p43	p43.1	28,140	28,765	28,715
	p43.2	28,480	29,265	29,170
	p43.3	28,835	29,545	29,535
	p43.4	83,005	84,110	83,985
ry48p	ry48p.1	14422	18,154	17,922
	ry48p.2	16,074	18,549	18,459
	ry48p.3	19,490	22,789	22,853
	ry48p.4	31,446	38,235	37,679
ft53	ft53.1	7531	8852	9056
	ft53.2	8026	9839	9588
	ft53.3	10,262	12,598	12,594
	ft53.4	14,425	17,650	16,935
ft70	ft70.1	39,313	43,460	42,707
	ft70.2	40,101	44,841	44,499
	ft70.3	42,535	48,015	48,311
	ft70.4	53,530	60,049	59,275

shows all the results obtained during the execution of the experiments.

Based on the results shown in Table 10, it can be seen that the final distance achieved by the proposed transfer learning system showed better results in several of the instances tested, more precisely in 14 of the 18 instances (i.e., in approximately $78\%$ of the simulated SOP problems). Among these results, the final distance values obtained in instances ft70.4, ft70.1, ft53.4 and ry48p.4 stand out. In these four instances, there was a difference of more than 500 units in the values obtained when learning with the Auto_TL_RL system and without the Auto_TL_RL system. Consequently, it can be seen that by using the proposed Auto_TL_RL system, it was possible to obtain final distance values closer to those presented as the best known solution by TSPLIB compared with the results obtained when the Auto_TL_RL system was not used.

5. Comparison with Other Studies

This section presents a comparison of this proposal with other literature studies. For this, three papers that applied AI algorithms to solve the SOP were selected [37,39,80].

Table 11. Comparison of this proposal with different studies that applied algorithms for the sequential ordering problem’s solution.

		Proposed	[37]	[39]	[80]
Dataset	TSPLIB	🗸	🗸	🗸	🗸
	SOPLIB	–	–	–	🗸
Algorithm	Ant Colony System	–	–	🗸	–
	Particle Swarm Optimization	–		–	🗸
	Reinforcement Learning	🗸	🗸	–	–
Meta-learning	Hyperparameter Tuning	–	🗸	–	🗸
	Transfer Learning	🗸	–	–	–
	AutoML	🗸	–	–	–

shows a summary of this analysis.

The first point to emphasize is the relevance of the dataset used in this paper. In this sense, TSPLIB is frequently adopted in the literature when applying methods to combinatorial optimization problems, especially those related to the TSP and SOP.

The second criterion to be observed in Table 11 is the algorithm adopted to resolve SOP instances. Several traditional optimization techniques have already been used, such as the ant colony system [39] and particle swarm optimization [80]. It is noteworthy that the authors of [39,80] achieved important results in solving several TSPLIB instances, finding the best known solution in several simulated situations, thus showing great potential in these methods for solving combinatorial optimization problems based on the ATSP and SOP.

However, this paper did not aim to make a direct comparison with these conventional methods from the literature. The objective was to investigate and propose advances in reinforcement learning algorithms for solving combinatorial optimization problems. In this sense, it is noteworthy that, due to the best knowledge of these authors, this work is only the second paper that addresses reinforcement learning methods for solving the SOP.

For this aspect, the work in [37] made important advances in the application of RL and hyperparameter tuning for the SOP. For example, the work in [37] assessed that, in general, the SARSA algorithm outperforms the Q-learning method in the search for better SOP solutions. Following this line, this paper continued to investigate the possibilities for improving RL techniques for combinatorial optimization problems. The main advance of this paper in relation to the literature is the proposal of a transfer learning approach between the ATSP and SOP, providing a reduction in computational cost and optimization of the solution with RL algorithms. Furthermore, this paper innovated by proposing a new AutoML method, which is responsible for applying meta-learning and automatic TL between simulated combinatorial optimization problems.

6. Conclusions

The main objective of this work was to propose and evaluate the efficiency of a methodology for transfer RL between two combinatorial optimization domains: the ATSP and SOP. For this purpose, two approaches were taken. The first approach involved a general analysis of the transfer of learning. The second addressed the use of AutoML for transfer learning.

Based on the analysis performed, this work highlights certain aspects:

• Results visualization and interpretation of the impact on the final route distance results obtained with the transfer of learning between classical combinatorial optimization problems;
• Statistical analysis of the impact on the computational time and route distance results obtained by applying learning transfer between combinatorial optimization problems;
• Development of a methodology for learning transfer from the source domain (ATSP) to the target domain (SOP);
• Proposal of a methodology to perform learning transfer in an automated way;
• Proposal of an AutoML algorithm for transfer learning applied to combinatorial optimization problems with reinforcement learning.

In terms of analysis, the effects of using the knowledge base (QATSP) in the objective domain (SOP) were evaluated. The results obtained from the statistical tests show that, overall, adopting transfer learning led to the calculation of shorter routes in the SOP problems (TSPLIB). Furthermore, in 13 of the 14 instances simulated ($92.8\%$), the average computational time was lower in the experiments using the QATSP base. Results visualization made it possible to evaluate the differences in the behavior of the learning curves when the RL transfer matrix was used or not used.

The AutoML approach involved the development of the Auto_TL_RL algorithm. The Auto_TL_RL algorithm has a feature that makes it possible to identify the presence or absence of a knowledge base for a given problem. When the knowledge base already exists, the issue is executed directly. However, when a previous knowledge base is not available, a knowledge base is generated and stored for possible reuse in future situations. To accomplish this, concepts discussed in [79] were used, such as transfer learning and automated learning.

The transfer learning performed by the Auto_TL_RL algorithm produced better results in 14 of the 18 TSPLIB instances analyzed (approximately $78\%$). This demonstrates its efficiency and importance compared with traditional learning methods. It is also important to highlight the importance of automated transfer, since all the human and computational effort required to carry out this study was reduced by using an automated learning system.

In future work, it is expected to apply the proposed approach to other combinatorial optimization problems. For this aspect, it is important to observe the three criteria presented when carrying out transfer reinforcement learning across combinatorial optimization problems: (1) problems with the same objective function; (2) similar datasets; and (3) transfer from the simple domain to the more complex domain. Moreover, a forthcoming paper will analyze the transfer of hyperparameter tuning between the domains: $\alpha$, $\gamma$, $ϵ$ and the reinforcement function [12,37,68,76]. Moreover, the aim is to conduct the experiment with different instances. It is also intended to improve the Auto_TL_RL system in order to make it possible to apply the parameters used during the knowledge base generation stage, which showed the best results. Furthermore, future work will analyze the energy efficiency of the AUTO_TL_RL algorithm and its CO₂ emissions, since the computational time can hide particularities in energy consumption.