Per-Instance Algorithm Configuration in Homogeneous Instance Spaces: A Use Case in Reconfigurable Assembly Systems

Abstract

The physical capabilities of a reconfigurable assembly system (RAS) increase the agility and responsiveness of the system in highly volatile market conditions. However, achieving optimal RAS utilization entails solving complex optimization problems effectively and efficiently. These optimizations often define homogenous sets of problem instances. While algorithm configuration in such homogeneous contexts traditionally adopts a “one-size-fits-all” approach, recent studies have shown the potential of per-instance algorithm configuration (PIAC) methods in these settings. In this work, we evaluate and compare the performance of different PIAC methods in this context, namely Hydra—a state-of-the-art PIAC method—and a simpler case-based reasoning (CBR) approach. We evaluate the impact of the tuning time budget and/or the number of unique problem instances used for training on each of the method’s performance and robustness. Our experiments show that whilst Hydra fails to improve upon the default algorithm configuration, the CBR method can lead to 16% performance increase using as few as 100 training instances. Following these findings, we evaluate Hydra’s methodology when applied to homogenous instance spaces. This analysis shows the limitations of Hydra’s inference mechanisms in these settings and showcases the advantages of distance-based approaches used in CBR.

1. Introduction

A reconfigurable assembly system (RAS) has an open architecture that enables it to respond swiftly and economically to sudden market changes [1]. However, besides the physical capabilities of an RAS, effective production planning remains crucial to the productivity of the system [1,2]. Generally, the design problems involved in production planning are formulated separately in a hierarchical manner and formulated as independent combinatorial mathematical optimization problems [1,3]. The process of developing and implementing the different algorithms (using either heuristic and/or exact methods) to solve each mathematical problem is usually a time-consuming endeavor. Furthermore, algorithm design decisions and identification of the appropriate algorithm configuration settings are well-recognized bottlenecks that require ample expert knowledge [4]. During this process, developers will typically use a limited number of benchmark instances, “fixing” some of the configuration settings based on the performance observed on those benchmark instances and exposing only a few configuration settings to the final user, who will, in many cases, leave them at their default values [4,5].

However, achieving optimal algorithm performance for each unique problem instance could be crucial in unlocking its maximum potential. Previous research extensively highlights that an algorithm’s effectiveness on one particular problem instance does not ensure superior performance across all instances [4,6,7]. In fact, certain algorithms outperform others on specific instances while underperforming on others, demonstrating what is commonly known as “performance complementarity”. This phenomenon is widely observed across virtually all NP-hard decision and optimization problems [8], including those pertinent to production planning. Nevertheless, it is “generally unknown a priori which of several algorithms should be used to solve a given instance” [8]. This decision problem is referred to as the per-instance algorithm selection problem (PIAS), or as the per-instance algorithm configuration problem when selection is made over all valid configurations of a single configurable algorithm [8,9].

Nevertheless, up until very recently, the literature on PIAC has usually dealt with (highly) heterogeneous instance spaces, with state-of-the-art approaches to PIAC and PIAS such as ISAC [10], and Hydra [11,12] developed as efficient methods to work within such heterogeneous contexts. However, solving the production planning problem in a RAS often requires that a solution be found repeatedly for the same (parametrical) optimization problem, each time with different problem parameters [13]. This often leads to a seemingly homogenous set of problem instances with a similar (if not identical) number of constraints and decision variables. This is not unique to production planning, but also a theme in other planning environments commonly found in industry, such as daily routing of delivery trucks, internal transport planning using automated guided vehicles in manufacturing facilities [14], or container allocation for rail cars leaving a consolidation terminal [15].

Research on PIAC for homogenous instance spaces remains sparse. Consequently, a common approach when dealing with homogeneous instance spaces is based on a “one-size-fits-all” approach [8]. This involves searching and evaluating numerous configurations across a (hopefully) representative set of problem instances to identify the best-performing configuration on average for the benchmark set, with no guarantees for other problem instances not in that set [16,17]. Moreover, some authors have discovered that homogenous instance spaces may exhibit inherent heterogeneous characteristics, making them better suited for PIAC approaches [16]. Additionally, recent findings by [18] suggest that PIAC methods can yield substantial performance improvements even when dealing with highly homogeneous instance spaces.

Consequently, in this work, we aim to further explore the application of PIAC methods in highly homogenous instance spaces. As our use case, we will follow the integer quadratic programming (IQP) problem formulation of the process planning problem in an RAS, as described in [19], and solve it using Gurobi. Similarly to [18], we will compare two drastically different methods for PIAC, namely the state-of-the-art method Hydra and a simpler case-based reasoning approach. However, contrary to [18], we will use PIAC to identify the algorithm configurations that minimize computation time rather than improving solution quality. This paper is a strong extension of the conference paper [20] where we first investigated the potential of PIAC methods for production planning in an RAS. The current work further evaluates the impact of the tuning time budget and/or the number of unique problem instances used for training on each of the methods’ performance and robustness. In addition, we present a discussion on the reasoning for Hydra’s shortcomings when applied to homogeneous instance spaces. Finally, we discuss the implications of our findings on the implementation of PIAC methods in a production environment and on the adoption of algorithm configuration methods.

The rest of the paper is structured as follows. Section 2 sets the background and motivations for the current work. An overview of the PIAC methods used in this work can be found in Section 3. Next, Section 4 defines the relevant use case. The experimental methodology is defined in Section 5. The results of the experiments are presented and discussed in Section 6. Finally, an overview of the main conclusions and possibilities for future work are presented in Section 7.

2. Related Work

The per-instance algorithm configuration problem (PIAC) can be extended without loss of generality from the per-instance algorithm selection problem (PIAS), and as such, it can be defined as follows [8].

Definition 1.

Given a set, I, of instances of a problem, P, an algorithm, a, that solves P, the set, Θ, of all possible configurations of a, and a performance metric, $m:\Theta \times I\to R$, that measures the performance of a with configuration $\theta \in \Theta$ on instance i, find a mapping function, $S:I\to \Theta$, that maps any problem instance, $i\in I$, to a configuration, $\theta \in \Theta$, of the algorithm, a, such that the performance metric, m, is optimal.

Whilst major contributions in the field of PIAS have been made during the last few decades, the PIAC remains an open challenge. The main difference between both problems lies in the size of the selection space. In the PIAS literature, algorithms are to be selected from a set of a few hundred algorithms at most. PIAC research, on the other hand, deals, in most cases, with intractable (algorithm) configuration spaces, which arise from its combinatorial properties [8]. Consider [12], for instance, where IBM ILOG’s CPLEX is used to solve a mixed integer programming (MIP) problem. The algorithm, at the time of their study, exposed 74 different configuration settings in total that are tunable and can affect performance, for a total of over $4.75\times {10}^{45}$ possible configurations. Another example is Gurobi (which we use in the present study), which exposes a total of 57 configuration settings for a configuration space of over $2\times {10}^{34}$ possible configurations [21]. Vast configuration spaces are common when using commercial solvers to solve complex mathematical optimization problems and drastically add to the complexity of the PIAC.

As such, even though the PIAC remains a highly challenging subject, Algorithm Configuration research has gained lots of interest over the last two decades and promising results have been obtained on other, albeit less general, “variants” of the Algorithm Configuration problem; namely the Single-instance algorithm configuration problem (SIAC) and the per-set algorithm configuration problem (PSAC) [8,22]. In the SIAC (also referred to as instance-oblivious tuning [10], problem-independent algorithm configuration [12] or in Machine learning literature as hyperparameter tuning [8]), for instance, the aim is to explore the configuration space in order to identify the best algorithm configuration to solve a unique instance [17]. Several methods have been developed to directly address the SIAC. These methods generally consider the SIAC as a meta-optimization problem where the goal is to find a unique configuration ${\theta }^{*}\in \Theta$ that delivers optimal performance for the problem instance $i\in I$, but provides no performance guarantee for any other instance $i^{\prime }\in I$, if $i\ne i^{\prime }$ [23].

The PSAC, on the other hand, defines a “one-size-fits-all” approach to the algorithm configuration problem. In the PSAC, the aim is to find a unique configuration, ${\theta }^{*}\in \Theta$, with the best (aggregated) performance over the set, I, that describes a class of closely related instances [6,23]. However, as in the case of the SIAC, a lack of generalization arises as the main drawback: there are no guarantees on the performance of the unique configuration, ${\theta }^{*}$, on a different class of instances $I^{\prime }$. Furthermore, there may exist at least one instance, $i\in I$, for which the configuration ${\theta }^{\prime }\in \Theta$ may yield better performance than the aggregated optimal configuration, ${\theta }^{*}$ with ${\theta }^{\prime }\ne {\theta }^{*}$, a problem that becomes more relevant for larger class sizes, or for ill-defined (heterogeneous) classes [16,17].

To overcome the fuzziness that arises from an ill definition of the classes, some authors rely on feature embedding to automatically identify clusters of instances based on their similarity in the resulting feature space [10,17,24]. Several studies exist that deal with the identification of these features, at least for the most relevant problem types. These features can be used for both algorithm selection as well as algorithm configuration problems. Ref. [25] describes what is perhaps the most widely applied set of MIP problem features currently available in the literature.

The availability of features to model problem instances gave place to the first prominent incursions into the PIAC for vast configuration spaces, such as ISAC [10] or Hydra [11,12]. These methods make use of portfolios of complementary algorithm configurations $\{{\theta }_{I^1}^{*},{\theta }_{I^2}^{*},\dots \}$, where each configuration in the portfolio dominates over a given instance class $I\in \{I^1,I^2,\dots \}$. As such, the differences between portfolio-based approaches to PIAC lie in the way the portfolios are identified and the mechanism to map a given instance to its most similar instance class. Regardless of these differences, portfolio-based methods have achieved significant gains in performance on widely adapted (heterogeneous) benchmark sets. However, as some authors point out, this is not without significant limitations [17,18]. In fact, some of the recognized drawbacks of portfolio-based methods are as follows:

 (i) 

Once again, they are related to generalization, as it is possible to find unique configurations outside of the portfolio with better performance than the best “average” configurator of the class for individual instances within the class,

 (ii) 

the expensive Exploration of the configuration space, since each new configuration explored needs to be evaluated against all (or a representative set of) the available problem instances. This is nonetheless a very expensive approach that limits the set of configurations that can eventually be explored (although some arguments in favor of this approach appear in the literature as a countermeasure to over-tuning [10,16]).

 (iii) 

Consequently, it becomes rather expensive to augment the portfolio (if, e.g., a new instance class or a new configuration with possible performance gains is identified), since adding a new configuration would require its evaluation in a large number of instances.

In response to these issues, [18] proposed a PIAC method that makes use of deep embedding, and case-based reasoning [26] for prediction. As such, a repository or case base of training instances, algorithm configurations (identified by solving SIAC on each individual instance) and pertinent algorithm performance is constructed. At runtime, a look-up methodology assigns the algorithm to a new (unseen) problem instance, the best configuration found for the closest instance available in the case base. Regardless of the embedding mechanism used, the case-based reasoning approach described in [18] helps overcome (i) by solving SIAC for each individual instance available (rather than solving the PSAC on the entire set or on a given class, as is the case for portfolio-based methods). Consequently, the approach allows for the exploration of a larger set of possible configurations (cf. (ii)), and makes it easier to augment the model (cf. (iii)) thanks to the focus on individual instances. Additionally, the authors in [18] report better performance (up to 38% increase in performance) when compared with that of Hydra on a homogenous instance set. However, it is worth noting that the authors evaluate their method using the best value (cost) of the objective value found by the algorithm as the performance metric. This way of working is mostly relevant if a large majority of the problem instances times out, i.e., an instance cannot be solved to optimality within a certain time span. This is not necessarily applicable to all cases, in which a diverse mix of capped and non-capped instances may be available. In such cases, a more common approach (perhaps the most widely used approach in the algorithm configuration literature) is to use the penalized average runtime, or PAR-k, as the metric of performance. This means that the runtime of runs that exceed the cut-off time of t seconds is penalized by a factor of k (usually $k=10$). This is also of ample relevance to our use case, since production planning for agile and responsive settings, such as in an RAS, requires that planning is (re)defined as promptly and efficiently as possible whenever needed. As a result, minimizing the allocated computational budget (without compromising solution quality) to solve each problem instance of the production planning problem becomes critical [27].

The case-based reasoning (CBR) approach for PIAC has shown promising benefits and improvements in performance, as reported in [18]. These improvements are particularly notable when it is compared with other state-of-the-art approaches such as Hydra. Our interest lies in evaluating these performance metrics further. Specifically, we want to determine if the observed performance for CBR and portfolio (Hydra) methods for PIAC, as reported in [18], remains consistent when the performance metric used is an indication of runtime (PAR-10) rather than solution quality. We will apply the case of production planning in an RAS environment for this study. Specifically, we will adopt the formulation of the process planning activities for an RAS, as defined in [19]. This formulation presents the problem as an integer quadratic programming (IQP) problem, which we will solve using Gurobi [21]. Considering the RAS’s operation in a highly volatile market, we will explore the solution of multiple IQP instances. These instances differ only in the magnitude of demand for each product type over a consistent time period. Consequently, we obtain a set of homogeneous instances that share the same number of constraints and decision variables, and a combination of capped and non-capped instances (See Section 5.1).

Additionally, we will investigate the impact of the training size, as well as the budget allocated for the exploration of the configuration space on the performance of the PIAC methods, a discussion that, to the best of our knowledge, is currently missing in the literature on algorithm configuration where the focus is usually on maximizing the performance of PIAC methods on benchmark sets. Very rarely is the trade-off between expected performance and the necessary budget (in terms of training time) required to obtain such performance considered. However, further knowledge about these trade-offs is of the highest relevance for the adoption of PIAC methods in production environments.

It is worth noting that the implementation of PIAC methods requires that the configuration space is explored to identify and select the best configurations associated with the “training” problem instances at hand. This exploration is generally carried out by iteratively solving the PSAC or the SIAC for portfolio and non-portfolio PIAC methods, respectively. Considering the wide spread of optimization solver suites such as CPLEX and Gurobi, we are interested in using the self-tuning tool provided by Gurobi [21] for the exploration of the configuration space. Note that some authors report significant gains in performance over the CPLEX tuning tool by means of state-of-the-art approaches such as SMAC [28], GGA [29], or ParamILS [30]. Gurobi’s self-tuning tool, however, is more recent than that of CPLEX [31], and thus academic evaluation of Gurobi’s tuning tool performance has not yet reached a consensus. Nonetheless, several authors in operations research and combinatorial optimization report the use of Gurobi’s self-tuning tool. Ref. [32] found reductions in the CPU time of up to 84% when using the tuning tool on certain problem types. Similarly, Ref. [33] found that using Gurobi’s tuning tool on a given problem instance can help identify configurations that are up to 40 times faster than Gurobi’s default settings. However, both [32,33] found that the configurations found by the tuning process do not generalize well to other problem types. Note that in all cases where we found that Gurobi’s tuning tool was used, this was carried out based on a “one-size-fits-all” approach, which explains the limitations regarding generalization found in the aforementioned studies and the lack of consistency we could observe across the studies that make use of Gurobi’s tuning tool; whilst the authors of [32,33,34,35,36] found that the configurations found by the tuning tool were better than Gurobi’s default configurations in at least some instances, other authors such as those of [37,38,39,40] found no significant difference.

Finally, self-tuning tools such as that of Gurobi (or CPLEX) are immediately available to those using the solvers in the first place for the solution of their optimization problems, thus setting a lower entry barrier to those interested in applying more complex methods for algorithm configuration such as PIAC or presenting a motivation for the use of alternative state-of-the-art configurators such as SMAC or ParamILS, to name a few.

3. Methods

Implementing a PIAC method requires the identification of the best configurations associated with a set of “training” problem instances. These configurations, together with the set of training instances, are then used at an inference step in which one predicts the best configuration to deal with a new problem instance. As such, a PIAC method is characterized by (i) the way the configuration space is explored and (ii) the mechanisms used for inference. This section explores these aspects for each of the PIAC methods we aim to study, namely the CBR and portfolio (Hydra) methods.

3.1. Case-Based Reasoning Method for PIAC

3.1.1. Exploration of the Configuration Space

In this work, we will use Gurobi solver v10.0 [21] to solve each of the IQP problem instances that define process planning in an RAS, as formulated by [19]. Commercial solvers such as Gurobi are powerful state-of-the-art solvers used widely in academia and industry. Gurobi exposes 57 configuration settings in total that can affect performance when solving MIP problems. Most of these parameters are categorical (a few are continuous) with 3 to 4 categories. This results in a vast configuration space of over $2\times {10}^{34}$ possible configurations. Fortunately, Gurobi incorporates a “Self-tuning tool” that can easily be accessed through the available APIs or via the command line interface. Given the proprietary nature of the solver, full details on the operation of the algorithm behind the “Self-tuning tool” are missing (and are free to change from version to version). However, different communication material made available by Gurobi gives us some insights regarding the way it works:

• Adaptive capping, for instance, is one of the features reported by Gurobi [21] (it is also easy to identify the adaptive capping feature in action when tuning is focused on algorithm runtime). It is a common feature proved to increase search performance and can be observed in many state-of-the-art configurators such as ParamILS [30], GGA [29], and Irace [41].
• The summary of the best configurations found follows a Pareto frontier [21] that focuses on configurations with fewer perturbations over the base/default settings.
• It performs replications with different random seeds to obtain more robust performance estimates [21].
• It allows for a secondary performance measure to guide the configuration search for instances that do not solve to optimality [21]; in such cases, solution quality becomes more relevant than the runtime, and thus available options include the minimization of the optimality gap, the best solution found (that with the best objective value), or the best objective bound.
• It allows for distributed parallel tuning [21], e.g., to distribute the tuning task over different CPUs in a computer cluster.
• The definition of a minimum acceptable performance target [21] so that tuning can stop once a minimum predefined performance is reached.
• It combines expert knowledge by means of certain “rules of thumb” in order to guide the search in combination with a random search [42].

For our experiments, we will use Gurobi’s self-tuning tool on each instance (i.e., solving the SIAC problem each time), and the best 10 configurations for each instance will be saved for retrieval during inference using the CBR method. Each call of the self-tuning tool will be carried out with the tuning tool’s default settings, except for the following:

• TuneResults (this tuning tool setting defines the number of configurations that yield an improvement over the default Gurobi settings that are eventually reported by the tool, which we will set to 10);
• TuneTrials (this tuning tool setting defines the number of replications that are run for each configuration on each instance, which we will set to 3);
• To investigate the performance of the PIAC method in relation to the budget allocated for the exploration of the configuration space, we will change the value of the setting TuneTimeLimit (this tuning tool setting defines the CPU time allocated to the tuning tool) accordingly (see Section 5.1).

Note that according to Gurobi, the default configuration settings are chosen based on Gurobi’s own collection of thousands of models from their customer base. As such, Gurobi claims that the default settings “work well on a wide range of model types”. In addition, for some of the configuration settings, default enables them to be automatically updated based on model structure, or even during the solution process based on observed model performance [43]. In spite of this, we have found in the course of our experiments that proper selection of Gurobi’s configuration settings can lead to significant gains in performance (See

Table 1. Overview of collected samples.

Case Base Type	Case Base 1	Case Base 2	Case Base 3
Tuning budget per problem instance (s)	200	400	600
Total number of instances	1000	1000	1000
Total number of (instance, configuration) pairs	4753	3344	3305
Total number of unique configurations	720	1437	1724
CPU time in seconds a (default; best configuration)	(2.57; 2.19)	(2.57; 2.11)	(2.57; 2.05)
PAR-10 in seconds a (default; best configuration)	(20.82; 12.02)	(20.82; 11.10)	(20.82; 10.06)
Number of capped replications a (default; best configuration)	(2.03; 1.09)	(2.03; 1.00)	(2.03; 0.89)
Number of instances with no capped replications (default; best configuration)	(171; 508)	(171; 594)	(171; 650)

^a Per instance average.

). The performance gain is related to the time budget allocated to the self-tuning tool. In our case, using the best configuration found by the tuning tool instead of the default configuration settings can lead to reductions in CPU time and PAR-10 of up to $20\%$ and $52\%$, respectively. The observed performance gains highlight the suitability of the tool for the case at hand, and the potential of algorithm configuration methods in general.

3.1.2. Training and Inference

As in [18], the CBR method will consist of a repository of instance features, $i\in I$, algorithm configurations, $\theta \in \Theta$, and algorithm performance, $KPI(i,\theta )$, stored as ($i,\theta ,KPI(i,\theta )$) tuples (Figure 1). At runtime, the method assigns, for each new instance, the best configuration found for the most similar instance in the repository. This approach is also known as case-based reasoning [26] since it consists of a set of “cases” that are retained. Each case corresponds to a description of a past example with its performance and is represented by a tuple ($i,\theta ,KPI(i,\theta )$). The repository of cases is also referred to as a case base. Next to [18,26] is another successful implementation of the CBR method. The main differences between [26] and [18]—besides the fact that the method in [26] was developed in the context of algorithm selection rather than algorithm configuration—are that (a) [26] presents a portfolio-based method, whereas that in [18] is based on SIAC implementations on each problem instance to build the case base, and the method in (b) [26] was designed with the goal of identifying an optimal algorithm schedule, i.e., a schedule of algorithms that are to be run sequentially for a given problem instance, whilst the method in [18] can be used to either identify a unique “best” configuration or a set of “good” configurations to be run in parallel (this approach is, however, out of scope in this work). Finally, (c) [26] makes use of designed features and Euclidian distance calculated over that feature space to assess similarity between instances, whereas [18] makes use of Euclidean distance on a feature space determined via a deep embedding approach.

In this work, we will focus on a case-based reasoning method where the case base is built via multiple SIAC implementations on each (training) problem instance. Furthermore, in our use case, we consider problem instances that vary only in the magnitude of the demand for each product type over a fixed period of time. Consequently, we will consider a set of features that emerge from the demand that characterize and differentiate each problem instance (See Section 5.2). As in [26], we will determine the similarity between problem instances using the Euclidean distance calculated over the resulting feature space.

3.2. Portfolio Method for PIAC: Hydra

Hydra is based on a portfolio of configurations with complementary performance. To build the portfolio, multiple iterations of SMAC [28] are performed on the entire training set, meaning that the PSAC is solved at each iteration. A configuration complements a given portfolio if its performance dominates that of any configuration already in the portfolio over a significant—undefined—set of instances in the training set.

For inference, Hydra makes use of an empirical performance model [25] built upon the resulting portfolio to identify the best configuration (out of those in the portfolio) to solve a new problem instance. The empirical performance model is based on a set of $\left(\right|{\theta }^P|-1)·|{\theta }^P|/2$ pairwise weighted cost-sensitive random forests (RFs) [44], where $|{\theta }^P|$ denotes the size (i.e., the number of unique configurations) of the portfolio.

To build the portfolios, we follow the same procedure as in [18]. As such, the portfolio is built iteratively from the set of the best configurations found when building the analogous case base used in the CBR method, rather than via multiple SMAC operations as in [12]. At each iteration, configurations from the case base are considered in order of prevalence in the training set, resulting in an idealized portfolio of complementary configurations. As in [18], the portfolio-building process stops when new configurations fail to improve the portfolio. Figure 2 presents an overview of the portfolio-building process.

Contrary to the work in [18], we will evaluate all PIAC methods at all times using the same method of feature embedding. This allows us to keep our focus on the inference mechanisms and their suitability to homogeneous instance spaces. The resulting set of features is described in Section 5.2. Finally, Ref. [18] considers the use of “presolver” runs. Such runs are used to collect “probing” features and solve “easy” instances while the more difficult instances are solved in a later run. Once again, to focus our analysis on the inference mechanisms, we will not consider “presolver” runs (or associated probing or presolving features) for any of the PIAC methods.

4. Use Case Definition

We will consider the assembly context defined in [19]. Ref. [19] considers a make-to-order multi-product assembly line with reconfigurable cells, and aims to determine the optimal process plan in this context. The authors consider a process plan the definition of the set of resources to allocate for the production of a given demand, the physical location of these resources at a given point in time, and the routing of parts on the shop floor. Figure 3 illustrates the layout constraints of the assembly system here considered. The maximum configuration available consists of a maximum of three possible hexagonal cells (locations I1–I3). Each cell has the capacity for six side modules (locations I7–I24) and one central module (locations I4–I6). Intercellular transport is enabled by a conveyor belt that runs parallel to the cells, whilst intracellular transport is carried out by the central module located at each of the cells. Finally, the distance between the cells does not allow for the simultaneous use of location pairs, (I11, I13), (I12, I14), (I17, I19), and (I18, I20).

Additionally, we will consider a single type of central module that can transport all parts in a product family. The side modules exist in two types: value-adding side modules that perform assembly tasks, and non-value-adding side modules that work in combination with the central module to facilitate the movement of parts from the conveyor into and off the cell (thus always occupying locations I7–I9). Seven different value-adding side module types can perform different assembly tasks (we consider three types of assembly tasks {A, B, and C}) with different levels of flexibility (side modules that can perform one, two or three task types), and one single non-value-adding side module type. Each central module and value-adding side module has a corresponding skill time and skill cost. The assembly context here defined will be modeled following the same IQP formulation of the process planning activities defined in [19]. The IQP formulation in [19] is parametrical and includes parameters that define the sequence of assembly operations needed to assemble each product type, the operational cost incurred by each resource for a given assembly/transport task of a given product type, the acquisition cost of each resource type, and the demand of each product, for example, among others. Finally, we consider a product mix of three products {p0, p1, p2} in an assembly system subject to high volatility with respect to the demand of products, meaning that the costs, assembly times, and resource types available remain constant across problem instances.

5. Experimental Methodology

In this section, we discuss the methodology used for the collection of samples used in the construction of the case bases, the definition of the instance features selected, and the definition of the validation sets.

5.1. Sampling Procedure

Recall that we will focus on problem instances that differ in nothing else than the parameter corresponding to the demand of products of each product type. For the construction of the case bases, we will consider a set with 1000 problem instances in total. Each problem instance is differentiated by the total demand of products of each type. For each product type, total demand varies in the range [84, 140] for a hypercube of over 175 thousand problem instances. From this hypercube, 1000 samples are collected via 100 replications of a k-Latin hypercube design (LHD) [45] with k = 10. For each problem instance, we will use Gurobi’s self-tuning tool 3 times to create 3 different case base types. Each call to the tuning tool (and consequently each case base type) will be given a different time budget, namely 200, 400 and 600 s. Note that to limit the influence of random noise, for each (instance, configuration) pair, 3 replications (calls to the solver) will be made during the tuning process, each with a different random seed. The 10 best configurations (i.e., those with the shortest CPU time) obtained (if any) by each call of the tuning tool will be stored individually to build the 3 case base types. Finally, to obtain a good mix of capped and non-capped replications, any call to the solver to solve a problem instance will be capped at 3 CPU seconds. Note that all samples and computations performed in this work were carried out in a 2.90 GHz Intel Core I9-8950HK CPU with 6 cores (12 logical processors) and 3GB of RAM. An overview of the resulting case bases can be found in Table 1.

The samples collected are evidence of a diverse mix of capped and non-capped problem instances. Furthermore, the sample overview in Table 1 showcases the large potential gains in performance that could be achieved through proper algorithm configuration, both in terms of computation time (and/or PAR-10) as well as in the frequency of capped replications. Note that the expected number of capped replications increases if the cut-off time decreases. Furthermore, for the same tuning time budget, the number of configurations explored decreases if the cut-off time increases. Instance sets for which the vast majority of solver calls are capped are more suitable to PIAC methods where the performance metric is a function of the solution quality and not CPU time.

5.2. Instance Features

Each instance in the case base will be represented by a set of features that describe it. Here, it is important to note that, when dealing with MIP, a common approach is to make use of the widely applied shallow embedding for MIP problems (cf. [25]) used in, e.g., Hydra. In our case, however, we have a highly homogenous instance space to the point that all instances share the same number of constraints and variables. This would result in the vast majority of the features described presenting the same value for all instances (due to all instances being, in most general aspects, the same IQP problem) and only a handful of features providing any discriminating value. Those features are, nevertheless, some simple transformations of the demand parameter. Consequently, we studied the suitability of the resulting set of “shallow” features together with some other relevant transformations of the demand parameter. For this assessment, we made use of a 10-fold cross-validation procedure on all case bases in Table 1. The best set of features identified during this process corresponds to a 9-dimension vector in which the first 3 dimensions correspond to the total number of product units for each of the 3 product types, the 4th to 6th dimensions correspond to the total number of task types, A, B and C, respectively, that need to be performed to assemble all product units, and the last 3 dimensions represent the maximum number of A, B and C tasks that need to be performed per period across all time periods. The resulting set of features had a better or equal performance on the validation sets than the small set of features from [25] that still managed to add some discriminative power. Nonetheless, just as the feature set in [25], the resulting feature set can also be considered the result of expert knowledge. The resulting feature set will be used in all experiments independent of the PIAC method or case base type.

Finally, also note that we are not interested in the use of “presolvers” (a predefined set of solvers that are called, either sequentially or in parallel, at runtime before any feature is computed, in the hope that they can deal with very simple instances before any inference takes place), and as such, we also omitted from the analysis any of the “probing” or “presolving” features (features that are obtained by briefly running the solver for the given instance at runtime to extract characteristics from the path the solver takes in order to solve the problem) described in [25].

5.3. Evaluation Methodology

To evaluate the methods, we will select 20 replications of the k-LHD (i.e., 200 unique problem instances) out of the 100 replications (1000 instances) collected to perform all the model evaluations (See Section 5.1). Since we aim to study the influence of the case base size (relative to the number of problem instances evaluated), the instances used to build the case bases will be drawn at random (without replacement) from the remaining instances (800 instances) until the given size is reached. We will always perform 10 replications of the evaluation process with different random seeds. Additionally, all PIAC methods are evaluated using the same validation set.

For the evaluation of the PIAC methods, we will consider 3 main performance metrics: PAR-10, CPU time, and the number of timed-out replications. The metrics will be reported as a marginal reduction or a percentual reduction with respect to the same performance metric obtained when using Gurobi’s default configuration (or other PIAC method). As such, the average marginal reduction, $KP{I}_{def}^j$, in a given kpi, $KPI\left(j\right)$, observed when using the configurations obtained with a given PIAC method, j, instead of Gurobi’s default configuration is calculated as follows:

$$KP{I}_{def}^j=\frac{1}{10}\sum _{r=1}^{10}KPI{\left(def\right)}_r-KPI{\left(j\right)}_r,$$

(1)

where $r\in [1,10]$ denotes the individual replications of the evaluation process, $KPI{\left(j\right)}_r$ denotes the performance metric observed when using the configurations obtained with the PIAC method, j, in the replication, r, of the evaluation process, and $KPI{\left(def\right)}_r$ denotes the performance metric observed when using Gurobi’s default configuration. Analogously, the average percentual reduction, $KPI{\%}_{def}^j$, in a given kpi, $KPI\left(j\right)$, observed when using the configurations obtained with a given PIAC method, j, instead of Gurobi’s default configuration is calculated as follows:

$$KPI{\%}_{def}^j=\frac{1}{10}\sum _{r=1}^{10}\frac{KPI{\left(def\right)}_r-KPI{\left(j\right)}_r}{KPI{\left(def\right)}_r}.$$

(2)

Finally, all confidence intervals for $KP{I}_{def}^j$ and $KPI{\%}_{def}^j$ presented in the next section are calculated as a 95% t-interval of the average performance estimate. Note that normality was tested in all cases visually as well as via 3 different normality tests, namely the Lilliefors test [46], the Shapiro–Wilk test [47], and the Anderson–Darling test [48]. In the cases in which the observed samples did not appear to fit a normal distribution, the confidence intervals were calculated on the same samples transformed using the Box–Cohen method [49]. However, since some of the observed samples of the performance metric estimate failed to fit a normal distribution even after transformation via the Box–Cohen method, in such cases, we calculated the confidence intervals using the bootstrap method [50] with 9999 bootstrap replications. The use of either the Box–Cohen or bootstrap method for the determination of the confidence intervals will be reported whenever applicable.

6. Experiments and Discussion

In this section, we perform some experiments that will help us assess the performance of the PIAC methods under different conditions. More precisely, we aim to investigate the impact of the tuning time budget per instance (represented by the case base type), as well as the case base size (the number of instances used to build the case base). For the latter, we consider case bases with 100, 200, 400, 600 and 800 problem instances. We start by assessing the performance observed when using the CBR method for PIAC in comparison with that when using Gurobi’s default settings. Next, we compare the performance of the CBR method for PIAC with the portfolio-based method, Hydra. Lastly, we dive deeper into the performance of Hydra, and provide an explanation of the performance observed. An overview of the results obtained during the experiments can be found in

Table 2. Average reduction in performance metrics (marginal and percentual) when using the configurations obtained with a given PIAC method instead of Gurobi’s default configuration. Metrics are calculated per case base type and per case base size. Results are presented as a 95% confidence interval. Results where the performance metric reduction is significant appear in bold. ^a Confidence interval calculated on transform data using the Box–Cox method. ^b Confidence interval calculated using the bootstrap method.

Metric	Case Base Size					Case Base Type	Method
	100	200	400	600	800	Case Base 1	CBR method
Avg. PAR-10 reduction (s)	[1.16, 2.34]	[1.34, 2.43]	[1.26, 2.11]	[1.39, 2.16]	[1.43, 2.29]
Avg. PAR-10 reduction (%)	[5.54, 11.11]	[6.36, 11.49]	[6.01, 10.02]	[6.63, 10.21]	[6.81, 10.84]
Avg. CPU Time reduction (s)	[0.01, 0.05]	[0.00, 0.06]	[0.00, 0.04]	[0.00, 0.04]a	[0.00, 0.04]
Avg. CPU Time reduction (%)	[0.47, 1.89]	[0.02, 2.17]	[0.01, 1.39]	[0.07, 1.51]a	[0.13, 1.66]
Avg. red. in No. of timed-out replications	[0.13, 0.26]	[0.15, 0.26]	[0.14, 0.23]	[0.15, 0.24]	[0.16, 0.25]
Avg. red. in No. of timed-out replications (%)	[6.22, 12.43]	[7.22, 12.81]	[6.83, 11.24]	[7.54, 11.43]	[7.72, 12.14]
Avg. PAR-10 reduction (s)	[2.21, 3.15]a	[2.80, 3.71]	[2.81, 3.35]	[2.94, 3.64]	[2.99, 3.61]b	Case Base 2
Avg. PAR-10 reduction (%)	[10.51, 14.97]a	[13.35, 17.55]	[13.40, 15.86]	[14.07, 17.20]	[14.28, 17.09]b
Avg. CPU Time reduction (s)	[0.05, 0.08]	[0.04, 0.09]	[0.05, 0.09]	[0.05, 0.09]	[0.05, 0.09]
Avg. CPU Time reduction (%)	[1.81, 3.16]	[1.75, 3.59]	[1.78, 3.32]	[2.11, 3.45]	[1.93, 3.65]
Avg. red. in No. of timed-out replications	[0.24, 0.34]a	[0.31, 0.40]	[0.31, 0.36]	[0.32, 0.40]	[0.33, 0.39]b
Avg. red. in No. of timed-out replications (%)	[11.70, 16.62]a	[14.95, 19.51]	[15.00, 17.64]	[15.72, 19.14]	[15.97, 19.02]b
Avg. PAR-10 reduction (s)	[2.63, 4.13]	[3.35, 4.02]	[2.96, 3.95]	[3.58, 4.03]a	[3.24, 4.01]	Case Base 3
Avg. PAR-10 reduction (%)	[12.52, 19.56]	[15.96, 19.07]	[14.09, 18.74]	[17.09, 19.05]a	[15.48, 18.96]
Avg. CPU Time reduction (s)	[0.06, 0.11]	[0.06, 0.10]	[0.06, 0.10]	[0.06, 0.10]	[0.06, 0.11]
Avg. CPU Time reduction (%)	[2.50, 4.08]	[2.41, 3.90]	[2.20, 3.99]	[2.51, 4.02]	[2.40, 4.19]
Avg. red. in No. of timed-out replications	[0.28, 0.45]	[0.36, 0.44]	[0.32, 0.43]	[0.39, 0.44]a	[0.35, 0.43]
Avg. red. in No. of timed-out replications (%)	[13.90, 21.73]	[17.80, 21.24]	[15.73, 20.83]	[19.01, 21.21]a	[17.27, 21.05]
Avg. PAR-10 reduction (s)	[−3.91, −2.44]	[−3.41, −2.72]	[−3.17, −2.67]a	[−3.71, −2.80]a	[−3.18, −2.63]	Case Base 1	Hydra
Avg. PAR-10 reduction (%)	[−18.75, −11.48]	[−16.22, −12.92]	[−15.26, −12.65]a	[−18.33, −13.11]	[−15.17, −12.46]
Avg. CPU Time reduction (s)	[−0.24, −0.20]a	[−0.24, −0.18]	[−0.24, −0.22]	[−0.26, −0.22]	[−0.24, −0.21]
Avg. CPU Time reduction (%)	[−9.50, −7.58]a	[−9.18, −6.90]	[−9.46, −8.43]	[−10.02, −8.55]	[−9.51, −8.15]
Avg. red. in No. of timed-out replications	[−0.41, −0.25]	[−0.35, −0.28]	[−0.32, −0.27]a	[−0.38, −0.29]a	[−0.33, −0.27]
Avg. red. in No. of timed-out replications (%)	[−20.03, −12.00]	[−17.26, −13.70]	[−16.15, −13.18]a	[−19.53, −13.71]	[−15.99, −13.03]
Avg. PAR-10 reduction (s)	[−2.54, −1.08]	[−3.73, −1.08]	[−4.17, −1.62]	[−3.87, −2.56]	[−3.30, −2.55]a	Case Base 2
Avg. PAR-10 reduction (%)	[−12.11, −5.14]	[−17.75, −5.18]	[−19.90, −7.70]	[−18.35, −12.17]	[−16.46, −11.87]
Avg. CPU Time reduction (s)	[−0.20, −0.09]	[−0.26, −0.12]	[−0.29, −0.20]a	[−0.26, −0.21]a	[−0.25, −0.21]
Avg. CPU Time reduction (%)	[−7.73, −3.61]	[−9.94, −4.55]	[−11.31, −7.28]a	[−10.23, −7.90]a	[−9.71, −8.00]
Avg. red. in No. of timed-out replications	[−0.26, −0.11]	[−0.39, −0.11]	[−0.43, −0.16]	[−0.40, −0.26]	[−0.34, −0.26]a
Avg. red. in No. of timed-out replications (%)	[−12.81, −5.27]	[−18.87, −5.24]	[−21.12, −7.87]	[−19.54, −12.75]	[−17.43, −12.39]
Avg. PAR-10 reduction (s)	[−4.63, −2.65]	[−4.69, −3.86]	[−5.05, −3.45]a	[−4.61, −4.00]	[−4.62, −3.39]	Case Base 3
Avg. PAR-10 reduction (%)	[−22.11, −12.58]	[−22.41, −18.25]	[−23.97, −16.66]a	[−22.03, −18.94]	[−22.06, −16.03]
Avg. CPU Time reduction (s)	[−0.30, −0.22]a	[−0.30, −0.27]a	[−0.31, −0.27]a	[−0.31, −0.28]	[−0.31, −0.25]
Avg. CPU Time reduction (%)	[−11.67, −8.10]a	[−11.78, −10.30]a	[−12.21, −9.86]a	[−12.00, −10.78]	[−11.95, −9.48]
Avg. red. in No. of timed-out replications	[−0.48, −0.27]	[−0.49, −0.40]	[−0.52, −0.37]a	[−0.48, −0.41]	[−0.48, −0.35]
Avg. red. in No. of timed-out replications (%)	[−23.61, −13.33]	[−23.91, −19.43]	[−25.71, −17.45]a	[−23.49, −20.02]	[−23.50, −16.93]
Avg. PAR-10 reduction (s)	[0.26, 1.73]	[1.25, 2.24]	[1.78, 2.46]	[1.85, 2.72]	[1.82, 2.80]	Case Base 1	CBR-portfolio
Avg. PAR-10 reduction (%)	[1.23, 8.23]	[5.92, 10.65]	[8.48, 11.72]	[8.80, 12.91]	[8.68, 13.24]
Avg. CPU Time reduction (s)	[−0.03, −0.01]	[−0.03, 0.01]	[−0.03, 0.02]	[−0.02, 0.03]	[−0.02, 0.03]
Avg. CPU Time reduction (%)	[−1.31, −0.21]	[−1.22, 0.48]	[−1.11, 0.76]	[−0.82, 1.11]	[−0.68, 1.04]
Avg. red. in No. of timed-out replications	[0.03, 0.19]	[0.14, 0.25]	[0.20, 0.27]	[0.21, 0.30]	[0.20, 0.31]
Avg. red. in No. of timed-out replications (%)	[1.55, 9.45]	[6.87, 12.12]	[9.75, 13.31]	[10.08, 14.62]	[9.96, 14.98]
Avg. PAR-10 reduction (s)	[2.78, 5.35]	[2.96, 6.34]	[2.52, 6.14]a	[3.23, 5.88]	[3.85, 6.23]	Case Base 2
Avg. PAR-10 reduction (%)	[13.16, 25.58]	[14.06, 30.00]	[13.45, 29.01]a	[15.34, 28.01]	[18.31, 29.59]
Avg. CPU Time reduction (s)	[0.04, 0.11]	[0.03, 0.16]	[0.01, 0.15]a	[0.02, 0.10]	[0.04, 0.13]
Avg. CPU Time reduction (%)	[1.38, 4.15]	[1.08, 6.22]	[0.08, 4.90]b	[0.67, 3.97]	[1.40, 5.22]
Avg. red. in No. of timed-out replications	[0.30, 0.58]	[0.33, 0.69]	[0.28, 0.67]a	[0.36, 0.64]	[0.42, 0.68]
Avg. red. in No. of timed-out replications (%)	[14.78, 28.61]	[15.84, 33.36]	[15.13, 32.27]a	[17.38, 31.38]	[20.65, 33.01]
Avg. PAR-10 reduction (s)	[−0.22, 3.89]	[−0.48, 4.48] a	[−0.94, 3.50] a	[−1.40, 0.45] a	[−1.25, −0.10]a	Case Base 3
Avg. PAR-10 reduction (%)	[−1.08, 18.64]	[−2.58, 21.48] a	[−4.82, 16.94] a	[−6.84, 2.65] a	[−6.06, −0.38]a
Avg. CPU Time reduction (s)	[−0.05, 0.05] a	[−0.05, 0.11] a	[−0.05, 0.05] a	[−0.08, −0.02]a	[−0.07, −0.04]
Avg. CPU Time reduction (%)	[−1.95, 2.36] a	[−2.33, 4.43] a	[−2.31, 2.05] a	[−3.37, −0.47]a	[−2.63, −1.36]
Avg. red. in No. of timed-out replications	[−0.02, 0.42]	[−0.03, 0.48] a	[−0.08, 0.37] a	[−0.14, 0.02] a	[−0.13, −0.02]a
Avg. red. in No. of timed-out replications (%)	[−1.01, 20.87]	[−2.69, 23.87] a	[−5.20, 19.00] a	[−7.34, 3.07] a	[−6.54, −0.23]a

.

6.1. Evaluation of the CBR Method for PIAC

Figure 4 shows the average reduction in PAR-10 obtained during evaluation when using the CBR method instead of the default configuration, for different case base sizes the (number of unique problem instances in the case base) and for different tuning time budgets (case base types). We can see how the CBR can lead to reductions of up to 17.2% in PAR-10 when using a case base with 800 unique instances and a tuning time budget per instance of 600 s (Case Base 3). Surprisingly, a case base with as few as 100 instances can lead to reductions of up to 16% for the same tuning budget. In fact, increasing the case base size seems to have a smaller effect on performance than increasing the tuning time budget (Table 3). However, smaller case base sizes seem to result in higher performance variance and thus less robust predictions (Figure 5).

Figure 4. Average reduction in PAR-10 obtained when using the configurations obtained with the CBR method instead of Gurobi’s default configuration. Results are presented per case base type and per case base size. The doted lines connect the mean values.

Figure 4. Average reduction in PAR-10 obtained when using the configurations obtained with the CBR method instead of Gurobi’s default configuration. Results are presented per case base type and per case base size. The doted lines connect the mean values.

Table 3. Marginal increase in performance metrics when increasing the case base size, and when increasing the tuning time budget for the CBR method. Results are presented as a 95% confidence interval. Results where the marginal increase is significant appear in bold.

Metric	Avg. PAR-10 Reduction	Avg. CPU Time Reduction	Avg. Reduction in the Number of Timed-Out Replications
From 100 to 800 instances @ Case Base 1	[−2.34, 3.34]	[−1.41, 0.84]	[−2.49, 3.70]
From 100 to 800 instances @ Case Base 2	[−0.57, 5.02]	[−0.78, 1.40]	[−0.57, 5.55]
From 100 to 800 instances @ Case Base 3	[−1.53, 3.89]	[−1.08, 1.09]	[−1.62, 4.30]
From Case Base 1 to Case Base 2 a	[5.88, 7.07]	[1.50, 1.94]	[6.49, 7.79]
From Case Base 1 to Case Base 3 a	[7.70, 9.27]	[2.16, 2.63] b	[8.49, 10.21]
From Case Base 2 to Case Base 3 a	[1.54, 2.48]	[0.41, 0.72]	[1.69, 2.74]

^a Average across all case base sizes. ^b Confidence interval calculated on transformed data.

Table 3. Marginal increase in performance metrics when increasing the case base size, and when increasing the tuning time budget for the CBR method. Results are presented as a 95% confidence interval. Results where the marginal increase is significant appear in bold.

Metric	Avg. PAR-10 Reduction	Avg. CPU Time Reduction	Avg. Reduction in the Number of Timed-Out Replications
From 100 to 800 instances @ Case Base 1	[−2.34, 3.34]	[−1.41, 0.84]	[−2.49, 3.70]
From 100 to 800 instances @ Case Base 2	[−0.57, 5.02]	[−0.78, 1.40]	[−0.57, 5.55]
From 100 to 800 instances @ Case Base 3	[−1.53, 3.89]	[−1.08, 1.09]	[−1.62, 4.30]
From Case Base 1 to Case Base 2 a	[5.88, 7.07]	[1.50, 1.94]	[6.49, 7.79]
From Case Base 1 to Case Base 3 a	[7.70, 9.27]	[2.16, 2.63] b	[8.49, 10.21]
From Case Base 2 to Case Base 3 a	[1.54, 2.48]	[0.41, 0.72]	[1.69, 2.74]

^a Average across all case base sizes. ^b Confidence interval calculated on transformed data.

Figure 5. Coefficient of variation of the average reduction in PAR-10 obtained when using the configurations obtained with the CBR method instead of Gurobi’s default configuration. Results are presented per case base type and per case base size.

Figure 5. Coefficient of variation of the average reduction in PAR-10 obtained when using the configurations obtained with the CBR method instead of Gurobi’s default configuration. Results are presented per case base type and per case base size.

Performance gains are more limited from the point of view of CPU time reductions. In the best case, the selection of a configuration via the CBR method reduces CPU time by as much as 3.3% on average (Figure 6). Once again, increasing the case base size seems to have a smaller effect on performance than increasing the tuning time budget (Table 3). The drastic difference in performance observed between the PAR-10 and CPU time can be better understood when we look at the average reduction in the number of timed-out replications (Figure 7).

The CBR method results in a reduced number of timed-out replications per instance (Figure 7). The reduction can be as much as 19.78% and falls in line with the magnitude of the average reduction in PAR-10 (Figure 4). Just as the case for the reduction in PAR-10, increasing the case base size seems to have a smaller effect on performance than increasing the tuning time budget (Table 3). In our experiments we found that the difference in performance between the CBR method and Gurobi’s default configuration is significant for all case base types and sizes (Table 2). Once again, good performance is obtained with as little as 100 instances (for Case Base 3), however, the large variance observed during the evaluation may be an indication of a lack of robustness. An increase in model robustness seems to be linked to an increase in case base size. As we can observe, the coefficient of variation (CV) of the performance estimate (the average reduction in PAR-10) seems to decrease monotonically as the case base increases (Figure 5). The simplicity of the CBR approach, however, allows us to easily augment the model by simply drawing new instances and performing tuning on each of them, without the need for re-training of the predictive model. This, in combination with standard cross-validation approaches (e.g., K-fold cross-validation), could help practitioners in the better planning of their sampling activities via a better understanding of the accuracy–robustness trade-off.

The application of the CBR method for PIAC in the process planning use case for RAS seems to indicate that significant performance improvements can be achieved with such a simple approach in the face of highly homogenous instance spaces. Furthermore, since the model can be easily augmented, our findings suggest that a more directed approach can be used to tailor the sampling efforts with the aim of achieving a desired level of accuracy and robustness. This is, however, out of scope in this work and is left out for future research.

6.2. Evaluation of Hydra: A Portfolio Method for PIAC

As for the CBR method, we begin the assessment of Hydra’s performance by looking at the average reduction in PAR-10 that results from the configurations predicted by Hydra in comparison with that when using the default Gurobi configuration (Figure 8). The observed performance seems to indicate that Hydra does not improve over the default configurations for the use case at hand. Furthermore, contrary to the CBR methods, both the performance (Table 4) and robustness (Figure 9) of the model seem to worsen as we increase the tuning time budget.

Figure 8. Average reduction in PAR-10 observed when using the configurations obtained with Hydra instead of Gurobi’s default configuration. Results are presented per case base type and per case base size. The doted lines connect the mean values.

Figure 8. Average reduction in PAR-10 observed when using the configurations obtained with Hydra instead of Gurobi’s default configuration. Results are presented per case base type and per case base size. The doted lines connect the mean values.

Table 4. Marginal increase in average PAR-10 reduction when increasing the case base size, and when increasing the tuning time budget for Hydra. Results are presented as 95% confidence intervals. Results where the marginal increase is significant appear in bold.

Metric	Avg. PAR-10 Reduction
From 100 to 800 instances @ Case Base 1	[−1.78, 3.74] b
From 100 to 800 instances @ Case Base 2	[−9.33, −1.75]
From 100 to 800 instances @ Case Base 3	[−6.27, 2.86]
From Case Base 1 to Case Base 2 a	[−0.38, 3.07] b
From Case Base 1 to Case Base 3 a	[−6.63, −3.93] b
From Case Base 2 to Case Base 3 a	[−7.80, −4.70] b

^a Average across all case base sizes. ^b Confidence interval calculated on transformed data.

Table 4. Marginal increase in average PAR-10 reduction when increasing the case base size, and when increasing the tuning time budget for Hydra. Results are presented as 95% confidence intervals. Results where the marginal increase is significant appear in bold.

Metric	Avg. PAR-10 Reduction
From 100 to 800 instances @ Case Base 1	[−1.78, 3.74] b
From 100 to 800 instances @ Case Base 2	[−9.33, −1.75]
From 100 to 800 instances @ Case Base 3	[−6.27, 2.86]
From Case Base 1 to Case Base 2 a	[−0.38, 3.07] b
From Case Base 1 to Case Base 3 a	[−6.63, −3.93] b
From Case Base 2 to Case Base 3 a	[−7.80, −4.70] b

^a Average across all case base sizes. ^b Confidence interval calculated on transformed data.

Figure 9. Coefficient of variation of the average reduction in PAR-10 obtained when using the configurations obtained with Hydra instead of Gurobi’s default configuration. Results are presented per case base type and per case base size.

Figure 9. Coefficient of variation of the average reduction in PAR-10 obtained when using the configurations obtained with Hydra instead of Gurobi’s default configuration. Results are presented per case base type and per case base size.

The limited performance of Hydra in terms of the reduction in PAR-10 is consistent with the poor performance observed in terms of CPU time reductions (Figure 10) and the average reduction in the average number of timed-out instances (Figure 11). We observed how Hydra consistently failed to improve compared with Gurobi’s default configuration. (Table 2) We will explore the reasons for the poor performance of Hydra in the next section (Section 6.3).

6.3. Discussion

During our experiments, we observed that the CBR method outperformed Hydra in the prediction of better configurations to solve each unique problem instance. These results fall in line with the findings of [18]. However, and perhaps surprisingly, Hydra’s performance during our experiments was not only inferior to that observed with the CBR method, but it also failed to improve upon the performance obtained when using the default configuration. Furthermore, Hydra’s performance did not improve consistently with an increased tuning time budget (as was the case for the CBR method). As such, to explain the poor performance of Hydra, one may need to take a look at (i) the characteristics of the portfolios that are built for each tuning time budget and (ii) the size of the instance set, and the way Hydra predicts a configuration.

A close look into the resulting portfolios shows, for Case Base 3 (the case base with the worst performance), that the portfolio building process results in smaller portfolios (Figure 12) than those in the case of Case Base 1 and 2. Furthermore, for Case Base 3, the portfolio size decreases as the case base size increases. This contrasts Case Base 1 and 2, where we can see an increase in portfolio size for larger case bases. As such, the smaller portfolios built in Case Base 3 may not be able to capture the full complexity of the problem space, which may, at least in part, explain the poor performance of Case Base 3. Considering the way in which the portfolios are built (see Section 3.2), it is possible that the most prevalent configurations used to build the portfolios are all present and have similar performance (with respect to one another) in the same area(s) of the input space. Consequently, the portfolio building process converges before the complexity of the input space can be captured. Case Base 1 and 2 both result in portfolios that are of similar lengths across the board (Figure 12), which may help explain their similar performance (Figure 8). However, as we can see in Figure 13, the portfolio building process yields portfolios with more variance in terms of their composition for Case Base 2. In fact, for almost all case base sizes, each replication of the portfolio building process yields a unique portfolio. For Case Base 1 and 2, on the other hand, the composition of the portfolios becomes more uniform as the case base size increases. The larger performance variance observed in Case Base 2 (Figure 9) may then be attributed to the varying composition of the resulting portfolios across the 10 replications.

The other aspect that may explain the poor performance of Hydra in the present use case is the way Hydra makes use of the portfolio data to predict a new configuration. Recall that inference in Hydra considers a voting system in which a set of pairwise (i.e., for each pair of configurations in the portfolio) weighted cost-sensitive RFs predicts/votes in favor of one configuration out of each pair of configurations. As such, the configuration with the most votes is selected. Furthermore, for each trained RF, a weight proportional to the performance delta observed between each configuration pair is assigned to the training data. This methodology has proven effective in dealing with (highly) heterogeneous instance spaces where the configurations found in the portfolios are strictly dominant in significant regions of the instance space. However, in a homogeneous instance space, such as in our use case, we observed that configurations lacked the ability to be strictly dominant over a given region and presented significant performance variations at a granular level. In a further analysis of the training data for a given pair of configurations ($C1$, $C2$) in a given portfolio, we observed that the distribution of the number of timed-out replications was somewhat similar between the configurations (Figure 14). The color regions have different thicknesses, and the areas of transition between color regions show different levels of overlap, but the distribution of the regions follows a similar fashion. Nevertheless, the most concerning aspect is perhaps that both configurations have a large number of instances in common (Figure 15), with the areas where three replications time out being those of most concern. We can see in Figure 14 that, in that same area, configuration $C1$ has a large number of iterations with two, one or even zero timed-out replications. That entire set of instances where both $C1$ and $C2$ time out for the three replications has no impact during the training of the RF for the pair of configurations ($C1$, $C2$) (since the associated weight is zero). Nonetheless, that same RF may cast a vote that would prefer one configuration over the other, which may result in wrongly selecting a configuration with very poor performance and thus a very large error (in terms of PAR-10). In general, we observed significant performance overlap within the space of each configuration (cf. Figure 14) and between configurations in the portfolio (cf. Figure 15). In fact, a configuration from the portfolio may be dominant over a certain region but present the worst performance among those configurations in the portfolio for sparsed instances within their dominant region. In such cases, selecting the dominant configuration instead of the best configuration in the portfolio can lead to differences (percentual errors) of up to 2900% in PAR-10. In short, the RF methodology proposed in Hydra seems to have trouble dealing with the strong performance overlap observed in the homogenous instance space.

Furthermore, we can verify the impact of the inference mechanism of Hydra on performance by replacing the RF methodology with the nearest neighbor approach used in the CBR method. Since the CBR method makes use of all training data for inference instead of a summary of the data (as the RFs do in Hydra), small local groupings can be more easily identified. To investigate this further, we build a case base that only contains the portfolio data used to train the RFs in Hydra, and use the same similarity-based approach to inference used in the CBR method; we will refer to this method as the CBR portfolio method. Note that the aforementioned CBR portfolio method is in essence the same as that in in [26] but in the context of algorithm configuration instead of algorithm selection. Figure 16 shows the performance in terms of the average PAR-10 reduction observed when using the configurations predicted by the CBR portfolio method for PIAC instead of Gurobi’s default configuration. We observed a significant improvement in performance when replacing the RF prediction method from Hydra with the nearest neighbor approach for prediction used in the CBR-portfolio method. The effects attributed to the robustness and length of the portfolios observed in Hydra remain present for the CBR portfolio method, namely the large performance variance (Figure 9 and Figure 17) and the poor performance observed in Case Base 3. However, even for Case Base 3, the nearest neighbor approach to inference used in CBR approaches results in significantly better performance than the approach used in Hydra.

Additionally, the CBR portfolio method seems to outperform the CBR method, albeit slightly, when using Case Base 1 (Figure 18). The difference observed is only statistically significant for a case base with 400 or more unique problem instances, but it is only 2.43% ± 1.96% at best. For Case Base 2, the marginal performance improvement when using the CBR portfolio method instead of the CBR method is only statistically significant for a case base with 800 unique problem instances and corresponds to 8.25% ± 5.31%. As such, even though the CBR portfolio method may have the potential for better performance (than the CBR method), the lack of robustness of the portfolio methods highlights issues regarding the consistency of performance estimates, and the difficulty of augmenting the model (since new samples may in some cases completely redefine the portfolio, evaluating new configurations implies evaluating them in all available instances). All of these issues are critical barriers to the implementation of portfolio methods in production environments that may tip the scale in favor of the CBR method. The CBR method, after all, showcases significant improvements over the default settings, high flexibility that facilitates the augmentation of the model even after deployment, and much more robust performance, which makes it more predictable, thus facilitating the assessment of sampling efforts to achieve a given performance.

7. Conclusions and Future Work

In this work, we assessed the performance of methods to solve the per instance algorithm configuration problem (PIAC) in a highly homogeneous instance space. We performed our study in the context of the solutions to the process planning problem in a reconfigurable assembly system (RAS). For this, we followed the formulation of the process planning problem as an integer quadratic programming (IQP) problem and solve each problem instance using Gurobi. For this use case, we compared two drastically different methods for PIAC, namely the state-of-the-art method, Hydra, and a simpler case-based reasoning approach (the CBR method). We focused on PIAC implementations to identify the algorithm configurations that minimize computation time rather than improving solution quality. Furthermore, we assessed the impact on each method’s performance with respect to the tuning time budget and/or the number of unique problem instances used for training.

Our work suggests the suitability of simple PIAC methods such as the CBR method in dealing with homogeneous instance spaces. Our experiments show that the algorithm configurations provided via the CBR method can lead to reductions in PAR-10 (when compared with Gurobi’s default configuration) of up to 16% with a case base that contains as few as 100 unique problem instances.

In contrast to the simpler CBR method, Hydra—a PIAC state-of-the-art method designed for (highly) heterogenous instance spaces—struggles to reach the performance of Gurobi’s default configuration. Additional analysis of Hydra’s performance suggests that the reason for Hydra’s poor performance could be attributed, in part, to the way its underlying portfolios are constructed.

Another aspect limiting Hydra’s performance in homogeneous instance spaces is perhaps its inference mechanism. To test this, we evaluated the portfolios used in Hydra following the same inference policy used in the CBR method, resulting in a portfolio-based CBR method. The CBR portfolio method drastically improved comapred with Hydra, and even in some cases seemed to outperform the CBR method, albeit slightly. Further discussion concludes, however, that the high flexibility, comparative higher robustness, and good performance of the CBR method are key advantages of using it instead of portfolio methods, especially in dynamic production environments such as a RAS.

Finally, the execution of either of the PIAC methods requires extensive sampling, which provides a big deal of information about the problem at hand. Future work may deal with the incorporation of this knowledge into the feature embedding process for different PIAC methods, or in the portfolio construction process. In addition, future work can also assess the impact of different sampling strategies (for the definition of the set of training problem instances) on the performance and robustness of the PIAC methods.