A Survey of Advances in Landscape Analysis for Optimisation

Fitness landscapes were proposed in 1932 as an abstract notion for understanding biological evolution and were later used to explain evolutionary algorithm behaviour. The last ten years has seen the field of fitness landscape analysis develop from a largely theoretical idea in evolutionary computation to a practical tool applied in optimisation in general and more recently in machine learning. With this widened scope, new types of landscapes have emerged such as multiobjective landscapes, violation landscapes, dynamic and coupled landscapes and error landscapes. This survey is a follow-up from a 2013 survey on fitness landscapes and includes an additional 11 landscape analysis techniques. The paper also includes a survey on the applications of landscape analysis for understanding complex problems and explaining algorithm behaviour, as well as algorithm performance prediction and automated algorithm configuration and selection. The extensive use of landscape analysis in a broad range of areas highlights the wide applicability of the techniques and the paper discusses some opportunities for further research in this growing field.


Introduction
This survey is a follow-up from a previous survey published in Information Sciences journal in 2013 [1]. Back then, the field of fitness landscape analysis was not very active in the evolutionary computation community. The article states "despite extensive research on fitness landscape analysis and a large number of developed techniques, very few techniques are used in practice . . . It is hoped that this survey will invoke renewed interest in the field of understanding complex optimisation problems and ultimately lead to better decision making on the use of appropriate metaheuristics." [1]. The hope of renewed interest in the field of fitness landscape analysis has indeed been realised, evident in the increase in the number of published papers on the topic as well as the appearance of tutorials, workshops and special sessions dedicated to this topic at all the major evolutionary computation conferences.
One of the changes that has emerged in the last few years is that the notion of fitness landscapes has been extended to include new types of landscapes such as multiobjective fitness landscapes, violation landscapes, dynamic and coupled landscapes and error or loss landscapes in the context of neural network training. These notions are discussed in Section 2.
A number of new techniques for analysing landscapes have been developed and these are described as an extension to the original survey [1] in Section 3, followed by a summary of contributions related to sampling and robustness of measures.
Landscape analysis has been applied widely for different purposes from the understanding of complex problems and algorithm behaviour, to predicting algorithm performance and automated algorithm selection. Section 4 provides a survey of these applications and highlights the value of landscape analysis in addressing the challenge of solving complex optimisation problems and Section 5 discusses some ideas for further research in landscape analysis.

Beyond Fitness Landscapes
The notion of a fitness landscape was introduced by Sewell Wright [2] at a congress on genetics in 1932. He proposed an abstract two-dimensional contour plot of fitness values as an intuitive picture of evolutionary processes taking place in high dimensional space. Because fitness landscapes have been used in contexts beyond biological and computational evolution, the fitness metaphor is no longer generally applicable and many studies have opted for more general terms such as search space analysis, exploratory landscape analysis or just landscape analysis.
A fitness landscape was originally defined as consisting of three elements [3]: (1) A set X of potential solutions to the problem; (2) a notion of neighbourhood, nearness, distance or accessibility on X, and (3) a fitness function f : X → R. Replacing f with a more general objective function, these three basic elements can be used to describe landscapes in a wide range of contexts such as combinatorial optimisation, continuous optimisation, search spaces of programs (as in genetic programming) and so on. There are some contexts, however, where these elements differ further from the original definition and these are discussed in this section.

Multiobjective Fitness Landscapes
Multiobjective optimisation differs from single objective optimisation at both the objective level (multiple fitness functions for each conflicting objective versus a single fitness function) and the solution-level (set of Pareto optimal solutions in the multiobjective case versus a single optimal solution in the single objective case). Verel et al. [4] propose a definition for multiobjective fitness landscapes where solutions are solution-sets, neighbourhood is defined using set-level operators and fitness is defined using a multiobjective quality measure. This formulation presents a completely different landscape from the fitness landscapes of the individual objectives but is arguably a more meaningful landscape to analyse when objectives are conflicting.

Violation Landscapes
Malan et al. [5] introduced the notion of a violation landscape as an additional view to fitness landscapes for constrained search spaces. A violation landscape is defined using the same elements as a fitness landscape, but the fitness function is replaced by a violation function that quantifies the extent to which a solution violates the constraints defined on the problem. A violation landscape is therefore defined above decision variable space and provides an additional landscape view to the fitness landscape. The features of violation landscapes can be analysed in relation to fitness landscapes to better understand constrained optimisation problems. Technique 28 in Table 1 describes metrics for characterising violation landscapes in relation to fitness landscapes.

Dynamic and Coupled Fitness Landscapes
Most landscape analysis assumes that the environment is constant, but there are many problems where the landscape is dynamic. Two scenarios where the dynamics of landscapes have been studied include applications where the objective function changes over time (due to changes in the problem environment) and in the case of coevolution, where landscapes are coupled [6] and influence each other.
An early approach to dynamic landscape analysis involved using existing landscape analysis techniques and simply interspersing landscape changes at set intervals during the search space sampling [6]. As an alternative, Richter [7,8] proposed using coupled map lattices for constructing fitness landscapes and modelling landscape dynamics. This framework was used to characterise landscape features (such as ruggedness and epistasis) over time as well as features specific to dynamic landscapes, such as change frequency and dynamic severity. Yazdani et al. [9] proposed an online approach to dynamic landscape analysis using a multipopulation method. In their framework, metrics are calculated based on information collected by subpopulations tracking peaks for quantifying changes in the landscape (peak shift severity, height variance and fitness variance).
In the context of coevolution, there are two distinct landscapes to consider: the objective landscape (the view of the problem to be solved) and the subjective landscape (the coevolutionary algorithm's view on the problem) [10]. It has been suggested that failure of coevolutionary algorithms could be due to disassociation between these two landscapes [11]. For coevolutionary games, Richter [12] proposed a dynamic landscape model based on a fitness interpretation of player payoff and used existing fitness landscape analysis techniques to analyse the game dynamics.

Error Landscapes
Part of neural network (NN) training involves optimisation, where the task is to search for the weight values that minimise the error of the network model on the training data. Analogous to a fitness landscape, the error or loss landscape can be analysed to better understand the nature of a particular NN weight optimisation problem instance. There are, however, some aspects of NN weight optimisation that make it different from most black-box optimisation problems and this affects the way that landscape analysis can be done.
Some of the distinguishing characteristics of NN weight optimisation are: ultra highdimensional search spaces (even small NN models can have thousands of weights), expensive objective evaluation (evaluating the error value of a solution weight vector involves a full run through the training data set), unbounded search spaces (weights can theoretically take on any real value), the same solution can evaluate to different error values depending on the subset of data instances used for training and the availability of analytical gradient information of the objective function. In addition, there is the added complication that the NN training landscape may differ from the NN testing landscape. Choromanska et al. [13] theoretically and empirically analysed the loss surfaces of multilayered neural networks and found that the training and testing error became increasingly de-correlated with the size of the network. The implication of this is that finding the global optimum in the training loss landscape is of limited use, because it will most likely not correspond with the position of the global optimum in the testing loss landscape. This presents a different picture than with most other optimisation tasks, where the focus is usually on finding the global optimum within the landscape.
Dimension reduction techniques have been proposed for visualising portions of error surfaces [14,15]. For example, three-dimensional visualisations of error landscapes (https://losslandscape.com/) are based on samples of weight vectors that fall on a two dimensional plane (a slice through the multidimensional weight space). The plane is positioned to pass through a point in the search space around which the landscape is visualised (such as the weight vector at the end of a training run). These visualisations provide a limited view that may not match the experience of a training algorithm in the high dimensional weight space. In addition, a form of numerical characterisation of error landscapes would be more useful for further analysis.
Bosman et al. [16][17][18][19] applied and adapted standard fitness landscape analysis techniques to error landscapes. Studies include: the influence of search space boundaries on the landscape analysis [16], the influence of regularisation on error surfaces [17], the influence of architecture settings on modality of the landscape [18], and the effect of different loss functions on the basins of attraction [19].

Advances in Landscape Analysis
This section discusses contributions to landscape analysis in the form of recently proposed techniques as well as studies in sampling and robustness of landscape analysis. For a background to the concepts and terminology of fitness landscape analysis, the reader is referred to earlier surveys [1,20].

Techniques for Landscape Analysis
An earlier survey of fitness landscapes in evolutionary computation [1] identified 22 techniques for analysing landscapes, from genetic algorithm (GA)-deception proposed by David Golderg in 1987 [21] to accumulated escape probability by Xin Yao and coauthors in 2011 [22]. The aim of the original survey was to make the techniques more accessible to researchers by describing each technique in an understandable way and highlighting attributes affecting their implementation in practice. Table 1 continues where the previous survey left off, introducing a further 11 techniques starting with technique 23 as local optima networks (LONs) [23]. Although LONs were mentioned in the previous survey as a model for describing the structure of landscapes, they were not listed as a practical technique for analysis. Since then the LON model has been extended and has evolved into one of the most widely used landscape analysis techniques today. The techniques in Table 1 appear in chronological order by the year of the first publication and are described under the following headings: • When the original list of landscape analysis techniques was compiled, an attempt to classify the techniques was unsuccessful. This was because the possible dimensions on which to base a classification did not help to distinguish between techniques. For example, consider a distinction between local/global or exact/approximate landscape analysis techniques. Many techniques did not fit into either of these hard classes. Even if they did fit into one class, small adaptations would move them into a different class. Instead of a classification of techniques, the approach used was to highlight distinguishing features to assist practitioners in deciding which approaches could be applicable in different scenarios.
The choice of an appropriate technique (or set of techniques) to use should be guided by the nature of the problems to be analysed as well as the purpose of the analysis. The Assumptions field in Table 1 specifies whether the technique could be applied to the given problems, while the Focus and Result fields specify whether the aim is likely to be met by the technique. For example, say your aim is to understand why a particular algorithm is failing on a vehicle routing problem (a combinatorial problem). Technique 23 (LONs) could be applicable, since it is designed for discrete search spaces and has a focus on the global structure of landscapes, which is known to affect search behaviour. LONs could be generated for a range of problems to contrast the global structure of problems on which the algorithm was successful and those on which it failed, to hopefully shed light on possible explanations for why the algorithm is failing. In a similar way, Techniques 25, 29 (and 31 if the algorithm was population-based) could be used to provide a different view of the problem focusing on other aspects namely variations in gradients, local fitness patterns (and evolvability), respectively. Technique 28 would only be appropriate if the problem was modelled as a constrained problem, while Technique 32 would only be appropriate if the problem was modelled as a multiobjective problem. Techniques 24, 27 and 30 would not be applicable because they assume a continuous search space, and Techniques 26 and 33 would also not be appropriate because the problem scenarios are different (coevolution and neural networks, respectively). Requires a complete enumeration of a discrete search space. Later extensions are based on samples to produce approximate LONs [32,33] and are adapted for continuous spaces [34]. Description: A LON is a graph-based abstraction of the search space representing the global structure, where each node of the LON is a local optimum and edges between nodes represent adjacency of the basins of optima (the possibility of search transitioning from one local optimum to another). The LON model is also extended to multiobjective problems to form pareto local optimal solutions networks (PLOS-nets) [30,31]. More detail and resources on LONs can be found on the website: http://lonmaps.com.

Result:
A graph visualisation showing the connectivity between local optima. Metrics can also be extracted from LONs such as number of optima, size of basins of attraction, shortest path to the global optimum [35], as well as funnel metrics [36,37], and PageRank centrality [38].
Technique 24: Exploratory landscape analysis (ELA) by Mersmann et al. [39] with extensions [40]. Year: 2011 Focus: Low-level features based on small samples Assumptions: Assumes a continuous search space. Description: Based on a small sample of random solutions (using Latin Hypercube sampling), six classes of low level features are defined: (1) convexity, (2) y-distribution, (3) levelset, (4) meta-model, (5) local search, and (6) curvature. Features are estimations of attributes such as the probability of the objective function being linear, the skewness of the distribution of the function values, the accuracy of fitted meta models, the number of local optima identified by local search, estimated numerical gradient and so on. The standard ELA feature set was later extended to include features based on general cell mapping (GCM) [40], but these are currently limited to low-dimensional spaces. ELA is supported by an online package in R, called flacco [41,42]  Variation in gradient estimations across the search space Assumptions: Assumes a distance metric in solution space. Description: Based on a sample of solutions from a random Levy walk through the search space, the length scale (the absolute difference in fitness over the distance in space) is calculated for each pair of solutions in the sample. The length scale distribution is defined as the probability density function of length scales and is estimated using kernel density estimation on the sample of length scales. Result: Plot of length scale distribution and a single value for the estimated entropy of the length scale distribution. Similarity between the objective and subjective landscapes in coevolution Assumptions: Assumes a model of coevolution for fast evaluation of subjective and objective fitness values. Description: Given a sample of points in the search space and two coupled and codynamic landscapes: the objective landscape (the fitness landscape of the problem) and the subjective landscape (how the coevolution perceives the problem) landscape measures are defined to quantify differences between the landscapes. Result: Three numeric values at each generation, quantifying different aspects of similarity.  Evolvability of a population Assumptions: Assumes a population-based algorithm for sampling.

Description:
Given a population of solutions and the set of neighbours (from one iteration of the algorithm), two metrics are defined: (1) epp is the probability that a population will evolve and is estimated by calculating the proportion of neighbours that are fitter than the best solution of the current population, and (2) eap is the evolutionary ability of the population, which is a quantity that increases with the absolute fitness improvement of the neighbours and decreases with the fitness diversity of the population. Result: A single value evp which is defined as epp × eap, with a range of [0, +∞].
Description: Given a sequence of solutions obtained through random walks and adaptive walks, features of the walk are derived from the sequence as a whole as well as the neighbourhood of solutions in terms of dominance and hypervolume improvement by neighbours. Result: 26 numerical values representing local features (17 from random walk sampling and 9 from adaptive walk sampling).
Technique 33: Loss-gradient clouds by Bosman et al. [19]. Year: 2020 Focus: Basins of attraction in neural network error landscapes Assumptions: Requires the numeric gradient of the loss function. Description: A sample of loss values and gradient values is obtained based on a number of random, progressive gradient walks [53]. Stationary points in the sample are determined to be local minima, local maxima or saddle points based on local curvature derived from the eigenvalues of the Hessian matrix. Stagnant sequences on the walk are detected by tracking the deviation in a smoothing of the error. Two quantities are measured: (1) the average number of times that stagnation was observed, and (2) the average length of the stagnant sequence.

Result:
A two-dimensional scatterplot of loss values against gradient values (loss-gradient cloud) and two metrics to estimate the number and extent of distinct-valued basins of attraction.
The first two techniques described in Table 1, LONs and ELA, are well established and have been widely used in studies involving landscape analysis, with LONs mostly applied in discrete optimisation and ELA in numerical optimisation. The wide adoption of these two approaches has been facilitated by the availability of code and online resources.
The previous survey [1] included techniques for detecting variable interdependence in binary search spaces (epistasis) but pointed out the absence of equivalent measures for continuous search spaces. This need has been addressed in the introduction of two techniques for quantifying nonseparability in continuous spaces (techniques 27 and 30).
A number of the new proposed techniques for landscape analysis apply to very specific optimisation contexts, such as coevolution (technique 26), constrained optimisation (technique 28), multiobjective optimisation (technique 32) and neural network training (technique 33). This corresponds to the extension of the notion of fitness landscapes to other types of landscapes discussed in Section 2.

Sampling and Robustness of Measures
Most landscape analysis techniques are based on a form of sampling of the search space. When considering the effect of the sampling on the analysis, it is not only the size of the sample that affects the outcome but also the sampling strategy [54,55].
Saleem et al. [56] proposed a method for evaluating landscape metrics in terms of the ability of the metric to identify trends in ordered sets of problems with specific landscape properties. They found that although some metrics can estimate features reliably using a small sample size, there are others that are very sensitive to the size of the sample. Muñoz et al. [57] proposed an experimental methodology for evaluating the reliability of landscape analysis methods that considers aspects such as vulnerability, volatility, stability and sensitivity to sample size. They showed that some landscape measures are highly volatile and that there is evidence of the curse of modality, requiring the sample size to increase with the number of local optima, rather than the dimension.
From the literature surveyed it is clear that random sampling techniques that are biased or structured in some form are more effective than more pure forms of random sampling for landscape analysis. For example, in the discrete domain (in the context of quadratic assignment problems), sampling strategies such as neutral walks were found to provide more insight into predicting problem hardness than random walks [58] and LON samples based on iterated local search had more predictive power for heuristic optimisation performance than samples based on random snowball sampling [37].
Alternative approaches to traditional random sampling for landscape analysis in the continuous domain include Latin hypercube design sampling [59], progressive random walk sampling [46] and gradient-based walks for sampling error surfaces of neural networks [53].
Online sampling, where the samples are based on solutions encountered by an algorithm during search, has become a popular alternative to random sampling for landscape analysis [60][61][62]. Enhancements to online sampling for continuous spaces include introducing a mechanism for correcting the error produced by the sampling bias [63] and the introduction of path relinking [64].
In an experimental study, Muñoz et al. [65] explored the effect that function translations had on landscape measures and found that translations could cause abrupt and severe changes in the values of some metrics. Škvorc et al. [66] also found that a large number of ELA features [39] (technique 26 in Table 1) were not invariant to shifting and scaling. Finally, Scott and De Jong [67] found that some landscape measures are very sensitive to the presence of noise in the fitness evaluation and that the error is difficult to correct for in an efficient way.
In some real-world optimisation scenarios, the objective function is very expensive to evaluate (such as in simulation-based optimisation) and surrogate models are used to approximate the fitness function. Werth et al. [68] performed a preliminary investigation into landscape analysis on surrogate functions and found that the landscape features were more indicative of the surrogate model than the original landscape.

Applications of Landscape Analysis
Landscape analysis has gained acceptance as a method for not only understanding complex problems and algorithm behaviour but also to support the prediction of algorithm performance and as part of automated algorithm configuration and selection.

Understanding Complex Problems
Landscape analysis has become popular as an approach to understanding complex optimisation problems by characterising problems using measured landscape features. When using landscape analysis to characterise problems, one of the aims is to group problems into classes based on these features. This is based on the premise that algorithms may behave similarly on problems that are similar. A vast array of complex classic and real-world problems have been studied using landscape analysis. Some examples in the last ten years are listed under three categories: (1) benchmarks or random instances of classic optimisation problems, (2) real-world applications and (3) machine learning applications.
Many real-world problems can be reduced to variants of classic problems, such as the quadratic assignment problem or the travelling salesman problem. Studying large benchmarks or random instances of these classic problems using landscape analysis is useful as general insights gained should be applicable to scenarios containing these problems as subcomponents. Examples of the application of landscape analysis to the study of classic problems include: the quadratic assignment problem [69][70][71]; the maximum satisfiability problem [72,73]; permutation flow-shop scheduling [74][75][76]; packing problems [77,78]; travelling salesman problems [79][80][81][82]; the dense graph-colouring problem [83]; number partitioning problem [84]; vehicle routing problems [85]; and the travelling thief problem [86].
Since 2017 a new trend has emerged of landscape analysis applied in the context of machine learning. Examples include: analysis of weight search spaces in the context of neural network training for classification [98]; analysis of the feature selection problem for classification [99,100]; analysis of policy search spaces in reinforcement learning [101]; analysis of machine learning pipeline configuration search spaces [102]; and analysis of neural architecture search spaces for image classification [103,104].
In each of the studies listed above, landscape analysis provided different insights into problem classes and the nature of the difficulty for search algorithms. To illustrate the kind of benefits that landscape analysis can provide, one study is described in more detail. In the field of genetic engineering, Simoncini et al. [94] analysed a computational protein design problem. The most popular software program used for solving these problems performed well on some instances but poorly on others. Landscape analysis revealed structural differences between the landscapes of an instance that was successfully solved by the software and one on which the software failed. It was observed that the algorithm performed poorly on a landscape that consisted of several suboptimal funnels that were disconnected from the global funnel. In this way, landscape analysis could explain why some problem instances were harder to solve than others. With this kind of understanding, algorithms can be adapted to be more effective on particular classes of problems.

Understanding and Explaining Algorithm Behaviour
The introduction of new metaheuristic algorithms is commonly justified on the basis of competitive experimental results on a limited set of benchmark problems. One of the problems with this approach is that these studies provide no scientific understanding as to why the algorithm performs well on the given problems, and more importantly, no understanding of when the algorithm will perform poorly.
Landscape analysis of optimisation problems provides a mechanism for explaining algorithm behaviour and identifying classes of problems that are suited to particular algorithms. Studies that describe the use of landscape analysis for understanding algorithm behaviour include: explaining evolutionary algorithm behaviour in dynamic environments [8] and the dynamics in coevolutionary games [12]; understanding the behaviour of local search algorithms [105,106]; explaining performance differences between search-andscore algorithms for learning Bayesian network structures [107]; explaining the effect of different mutation operators [108] and different function sets [109] in genetic programming; explaining the performance of different real-valued evolutionary algorithms [87]; explaining the performance of multiobjective evolutionary algorithms [52,110]; understanding evolvability in grammatical evolution [111]; understanding the effect of funnels in the landscape on metaheuristic performance [112]; and explaining the performance of evolutionary algorithms in generating unit tests for software [93].
The list above shows that landscape analysis has mostly been used to understand different evolutionary algorithms, but there are examples where other types of algorithms have been studied, such as local search algorithms and search-and-score algorithms.

Algorithm Performance Prediction
Although most metaheuristics can be easily understood in terms of their algorithmic elements, the behaviour that emerges is often unpredictable. Landscape analysis can be used to extract general features of problems to be used as input to machine learning models for predicting algorithm performance. These models are an important component in the wider aim of automated algorithm selection. Examples of the application of machine learning to predicting algorithm performance based on landscape features include the following: • Bischl et al. [113] used one-sided support vector regression to predict the bestperforming algorithm from a portfolio of four numerical optimisation algorithms based on ELA features. They showed that the model was able to generalise on new problem instances and predict the optimal or close to optimal algorithm from the portfolio. • Muñoz et al. [114] used a neural network regression model to predict the performance of a CMA-ES algorithm based on landscape features and algorithm parameters. Per-formance was measured in terms of the number of function evaluation required and they found that the model was able to predict the relative ranking values for given algorithm-parameter combinations effectively. • Malan and Engelbrecht [115] used decision tree models to predict failure of seven variants on the particle swarm optimisation algorithm based on landscape features. The models of five of the algorithm variants achieved testing accuracy levels above 90%. • Liefooghe et al. [30] used a random forest regression model to predict the performance of multiobjective optimisation algorithms in combinatorial optimisation based on a combination of landscape features and problem-specific features. They later developed a decision tree model for selecting the best performing algorithm out of three multiobjective algorithms [50]. Their model was able to predict the best performing algorithm in more than 98.4% of the cases. • Jankovic and Doerr [116] proposed a random forest regression model for predicting the performance of CMA-ES algorithms based on ELA features in a fixed-budget setting. They obtained high-quality performance prediction by combining two regression models trained to predict target precision and the logarithm of the target precision. • Thomson et al. [117] used random forest and linear regression models to predict algorithm performance for solving quadratic assignment problems based on landscape features derived from LON sampling. They found that random forest trees performed better at prediction than linear regression.

Automated Algorithm Selection
Performance complementarity is a phenomenon where different algorithms perform the best on different types of problem instances [118]. Automated algorithm selection is the process of deciding on the best algorithm for the problem at hand and depends on having features for distinguishing problems from each other. The features can be problem specific but can also be derived through landscape analysis, which can be more generally applied across problem types.
Algorithm configuration (choosing of algorithm parameters and strategies for particular problems) is closely related to algorithm selection but differs in that the set of possible configurations is much larger than a finite choice of algorithm candidates. The features used for algorithm configuration and selection can, however, be the same, so studies related to algorithm configuration are also mentioned here. Examples of algorithms that have been configured or dynamically adapted based on landscape characteristics include: • genetic algorithms: using the fitness distance correlation landscape measure to dynamically adjust the migration period in a distributed genetic algorithm [119], selecting a crossover operator based on fitness landscape properties [120], using fitness landscape features to estimate the optimal population size [121]; • differential evolution algorithms: adapting the strategy and adjusting the control parameters based on detected landscape modality [122,123], adapting the mutation strategy based on landscape features [124,125], algorithm configuration based on exploratory landscape features with an empirical performance model [126]; • memetic algorithms: analysis of the separability of problems to automatically select operators [45] and the use of four fitness landscape analysis techniques to inform the most suitable crossover operator [127]; • selection of CMA-ES algorithm configuration using a trained model for predicting performance based on landscape features that was shown to outperform the default setting of CMA-ES [128]; • surrogate-assisted particle swarm optimisation, where fitness landscape analysis was used to select surrogate models [129]; and • decomposition-based multiobjective evolutionary algorithms (MOEA/D), where the addition of landscape information improved the behaviour of the adaptive operator selection mechanism [130].
There are many attempts at automated algorithm selection using landscape features and the reader is referred to a survey by Kerschke et al. [118] for an overview of studies. Recent contributions not included in the Kerschke et al. survey include algorithm selection for the quadratic assignment problem [131,132], algorithm selection for the travelling salesman problem [133], algorithm selection for the permutation flowshop problem [134] and a form of automated algorithm selection for constraint handling techniques with differential evolution [61].
From the above, it is clear that landscape analysis is playing an important role in both algorithm performance prediction and automated algorithm configuration and selection.

Opportunities for Further Research
This survey highlights that fitness landscape analysis techniques are being applied in contexts beyond evolutionary computation. The wider scope of landscape analysis introduces interesting challenges and offers many opportunities for further research. Three ideas are discussed here.
Technique 26 in Table 1 proposes an approach to analysing the search space of coevolution by tracking differences between the objective and subjective landscapes. This notion of coupled landscapes, where landscapes are dynamic and a change in one effects the other, may be applicable in contexts wider than coevolution. For example, in generative adversarial networks (GANs), the error landscapes of the generator and discriminator networks are coupled in a similar way to the landscapes of coevolution. It would be interesting to investigate whether landscape analysis could be used in the context of GANs to better understand the dynamics of adversarial training.
Most of the research in landscape analysis is restricted to single objective search spaces. However, many real-world problems have multiple conflicting objectives. Technique 32 in the survey [50] is an important contribution as it provides a first set of numerical features for characterising local features of multiobjective problems. More work is needed in applying this approach to a wider range of problems and algorithms and also adapting the approach for use in continuous search spaces. There is still a gap in techniques for practically characterising global features of multiobjective search spaces. PLOS-nets [30,31] have been proposed for capturing the global structure of multiobjective landscapes, but it is still not clear how this approach can be scaled to large-size problems.
Surrogate modelling has become an important technique for managing optimisation problems that have computationally expensive objective functions. Initial investigations into landscape analysis of surrogate functions [68] were not very successful and further work is needed to identify landscape analysis techniques that are suitable for characterising surrogate functions so that the analysis is indicative of the characteristics of the actual landscape.

Conclusions
Research in landscape analysis has moved from being a theoretical topic in evolutionary computation to being extensively applied as a practical tool in the wider context of optimisation and has recently also been applied in machine learning. This survey describes advances in landscape analysis in the last decade, including a number of new techniques for landscape analysis and studies relating to sampling and robustness of measures. The survey also highlights the wide range of applications of landscape analysis in understanding complex problems, explaining algorithm behaviour, predicting algorithm performance and automatically configuring and selecting algorithms. Landscape analysis clearly has an important role to play in reducing the unpredictability of algorithms and advancing the field of optimisation and machine learning to a place where our technology can be trusted to solve real-world problems.

Conflicts of Interest:
The author declares no conflict of interest.