Calibration of an Adaptive Genetic Algorithm for Modeling Opinion Diffusion

Round 1

Reviewer 1 Report

The paper describes a Genetic Algorithm (GA) for solving problems based on the DeGroot opinion diffusion model. Special attention has been paid to control parameters (hyperparameters). The novelty and significance of this work are rather low. Furthermore, it seems that authors are unaware of recent developments in the field of Evolutionary Computation. However, I would like to give the authors a chance to improve the experimental part and writings.

1) Introduction is rather short on motivation, as well as on objective. Both should be better described. Why do other state-of-the-art metaheuristics also fail?

2) The authors claim that control parameters (hyperparameters) govern GA behaviour. To a large extend, this is true. But, the authors fail to recognize that the cornerstones of any search algorithm are exploration and exploitation. With a different setting of control parameters (hyperparameters), metaheuristics are implicitly controlling exploration and exploitation. There are some good surveys on the exploration and exploitation of evolutionary algorithms, which cover this topic more deeply.

3) The name of the hyperparameter “chromosomes”, which stands for the number of chromosomes, is odd. The usual term in GA is pop_size (population size). It indicates that the authors are not very familiar with the terminology and literature in this field. They are probably working in other fields.

4) This work is not replicable since the authors didn’t even provide pseudocode of the proposed GA, which is only described prosaically. The link to the actual code would be beneficial, as well. Furthermore, reintroducing elite chromosomes are even not discussed.

5) The proposed GA is adaptive GA. But, the authors failed to recognize it. There is no comparison with other adaptive Evolutionary Algorithms (EAS), and a precise description of adaptation is not given (should be clearly seen in the pseudocode).

6) Literature on Evolutionary Algorithms (and particularly GA) is either old or non-existent. Fundamental references on exploration and exploitation, adaptive parameter control, parameter tuning, and statistical comparison of EAs are missing.

7) It seems that authors are not aware of tuners (e.g. CRS-Tuning, F-Race, REVAC), which are typically used for setting control parameters (hyperparameters). At least some discussion is in order.

8) The experimental part with different population sizes (chromosomes) (5, 21, 51, 99) is flawed when using the maximum number of generations as a stopping condition. In such a case, the number of fitness evaluations consumed is very much different for various population sizes (e.g., with 100.000 generations and population size 5 there are 500.000 fitness evaluations; with the population size 99, there are 9.900.000 fitness evaluations). Obviously, the results (RMSE) should not be compared under such unfair conditions.

9) NHST (Null Hypothesis Significance Testing) has not been applied, and authors’ haven’t shown that a particular setting of control parameters (hyperparameters) is indeed statistically significant.

10) Discussion section is weak on how these results can be generalized if only one sample of Erdös–Rényi network has been used. What are the threats to validity?

11) Conclusion section has been completely omitted.

Author Response

1) Introduction is rather short on motivation, as well as on objective. Both should be better described. Why do other state-of-the-art metaheuristics also fail?

• We clarified the meaning behind our goal of making the algorithm more accessible and how this is accomplished through the simulation study in the Introduction. We also added information on why existing algorithms for modeling opinion diffusion were insufficient and justified our selection of a genetic algorithm over other bio-inspired algorithms.

2) The authors claim that control parameters (hyperparameters) govern GA behaviour. To a large extend, this is true. But, the authors fail to recognize that the cornerstones of any search algorithm are exploration and exploitation. With a different setting of control parameters (hyperparameters), metaheuristics are implicitly controlling exploration and exploitation. There are some good surveys on the exploration and exploitation of evolutionary algorithms, which cover this topic more deeply.

• We make the distinction between the parameters directly controlling the behavior of operators (control parameters) and the parameters governing the overall behavior of the algorithm (hyperparameters) in the Other Features subsubsection. We do not believe the statement that these parameters govern GA behavior is inaccurate and address the nuances of adaptation of the control parameters to balance exploration and exploitation once the necessary information for this discussion has been presented. We previously discussed the concepts of exploration and exploitation in the description of the operators as well as the Results and Discussion sections using the terminology of “exploration” and “refinement”. We have rephrased all relevant sections to use “exploitation” in addition to or instead of “refinement” to clearly indicate the connection to this area of the literature. We also added a reference to this concept before describing the operators with relevant citations and included this terminology elsewhere to highlight other areas related to exploration and exploitation.

3) The name of the hyperparameter “chromosomes”, which stands for the number of chromosomes, is odd. The usual term in GA is pop_size (population size). It indicates that the authors are not very familiar with the terminology and literature in this field. They are probably working in other fields.

• Since this method was developed for social science and public health research, “population” could be confused with either the individuals in the social network or population of interest. While atypical, our use of “chromosomes” should be clear regardless of research area, and its meaning is defined in Table 1.

4) This work is not replicable since the authors didn’t even provide pseudocode of the proposed GA, which is only described prosaically. The link to the actual code would be beneficial, as well. Furthermore, reintroducing elite chromosomes are even not discussed.

• A link to the full code is already included in the Data Availability Statement. We added a reference to code availability in the body of the manuscript. We mention that we use elite chromosome reintroduction in the text and table at the beginning of the Hyperparameters subsubsection. We use the phrasing “number of generations without improvement before the control parameters are modified and the elite chromosome reintroduced” at the beginning of the Parameter Recovery subsection before switching to the shorter “number of generations without improvement”. The relevant hyperparameter (iterr) is included with all others in this group and this set of hyperparameters is discussed throughout the Results and Discussion sections. We updated one instance of “number of generations without improvement before the control parameters are modified” to include the chromosome reintroduction.

5) The proposed GA is adaptive GA. But, the authors failed to recognize it. There is no comparison with other adaptive Evolutionary Algorithms (EAS), and a precise description of adaptation is not given (should be clearly seen in the pseudocode).

• We added the “adaptive” terminology to highlight this feature of the algorithm, including it specifically in sections where the adaptive nature was previously described using different language. We incorporated a discussion of other bio-inspired algorithms and justify our choice of a genetic algorithm in the Introduction section. The description of the adaptation is in the Other Features subsubsection, and we incorporated a specific example to demonstrate how this multiplicative adjustment is implemented. The specifics of the implementation of the multiplicative adjustment can also be seen in the code.

6) Literature on Evolutionary Algorithms (and particularly GA) is either old or non-existent. Fundamental references on exploration and exploitation, adaptive parameter control, parameter tuning, and statistical comparison of EAs are missing.

• We previously included references covering the concepts of both balance between exploration and exploitation and adaptive parameter control but have included additional citations along with our incorporation of this language. Initial citations for parameter tuning primarily focused on the importance of parameter tuning and the potential for this to be a barrier to use, but we have incorporated literature on various tuners along with our justification of the use of a simulation study instead of a more sophisticated tuner at the beginning of the Algorithm Calibration subsection. Background on statistical comparison of EAs is excluded since a comparison to other algorithms is not relevant to our intended purpose of making our methodological development more accessible.

7) It seems that authors are not aware of tuners (e.g. CRS-Tuning, F-Race, REVAC), which are typically used for setting control parameters (hyperparameters). At least some discussion is in order.

• We incorporated both citations and a discussion of the use of a simulation study instead of these tuners in the Algorithm Calibration subsection. We use a simulation study due to its simplicity, accessibility to researchers across fields, flexibility to provide different suggestions across network and dataset features, and ability to assess the impact of various hyperparameters instead of providing a set or sets of ideal hyperparameters.

8) The experimental part with different population sizes (chromosomes) (5, 21, 51, 99) is flawed when using the maximum number of generations as a stopping condition. In such a case, the number of fitness evaluations consumed is very much different for various population sizes (e.g., with 100.000 generations and population size 5 there are 500.000 fitness evaluations; with the population size 99, there are 9.900.000 fitness evaluations). Obviously, the results (RMSE) should not be compared under such unfair conditions.

• While we use number of generations as a stopping condition, it is not a hyperparameter we investigate in this study and was used to simplify the structure of the dataset. Since the vast majority of runs identify a solution before this threshold is reached, we do not believe this is of concern. Our use of RMSE without accounting for the number of fitness evaluations would be an issue if we were performing statistical tests, but the RMSE is not used in this way. When assessing hyperparameters other than the number of chromosomes, we present the data so that each value of number of chromosomes is equally represented in each group. When assessing number of chromosomes, we highlight that each generation will require more time when more chromosomes are used. It is worth noting that number of fitness evaluations would have similar flaws when assessed across different network and dataset sizes.

9) NHST (Null Hypothesis Significance Testing) has not been applied, and authors’ haven’t shown that a particular setting of control parameters (hyperparameters) is indeed statistically significant.

• We clarified in the Introduction section that our goal is to provide a set of suggested default or initial values where possible and present the information necessary to understand behavior and make informed choices and modifications otherwise. A statistically significant difference in performance for a set or sets or hyperparameters does not further this goal. We discuss statistical significance as is relates to other tuning methods in material added to the Algorithm Calibration subsection.

10) Discussion section is weak on how these results can be generalized if only one sample of Erdös–Rényi network has been used. What are the threats to validity?

• We generate a new network for each run of the algorithm (10 runs for each combination of inputs in Table 5). We clarified this point and justified the choice of Erdös–Rényi networks in the Procedure subsubsection.

11) Conclusion section has been completely omitted.

• The conclusion is omitted per the MDPI instructions that the conclusion section is optional and can be included if the discussion is unusually long or complex.

Reviewer 2 Report

The paper could be interesting but needs a major revision. My comments on the paper are as follows:

1. I do not understand why the article 'Performance...... Behavior Interventions' is not cited in the current paper. It gives the impression that the authors could have included the current paper at

https://www.mdpi.com/1660-4601/18/24/13394 

Why? What does the article under review add to the work cited here by the same authors?

2. It would also be interesting for the authors to justify the use of GAs with respect to other heuristic optimization techniques in the study of social network opinion models, for example:

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1078.3250&rep=rep1&type=pdf  

3. In the end the study of hyperparameters does not seem so interesting since finally in section 3. Results it is said that the algorithm found the solutions without taking into account the value set to the hyperparameters in 67.7% of the experiments, which suggests that the evolutionary surface for the DeGroot opinion model should not be very complex, is that so? Please see: https://link.springer.com/article/10.1007/s11042-020-10139-6)

4. why haven't you compared GA with other heuristic optimization techniques? For example: Particle Swarm Optimization, Bacterial Foraging, Simulated Annealing, etc. Please see this article although there are many examples in the literature:

https://www.researchgate.net/publication/271832435_PV_parameter_estimation_using_different_evolutionary_algorithms

5. You use parameter recovery, why not use other techniques? Please explain the advantages of your protocol in the DeGroot model or opinion pramatros.

Author Response

1. I do not understand why the article 'Performance...... Behavior Interventions' is not cited in the current paper. It gives the impression that the authors could have included the current paper at

https://www.mdpi.com/1660-4601/18/24/13394 

Why? What does the article under review add to the work cited here by the same authors?

• While relevant, this work was not yet accepted and was not cited for this reason. We have added the relevant information and appropriate citations. The work presented in both the previous publication and current manuscript use the genetic algorithm but have distinctly different goals and analyze data from different simulation studies. The IJERPH article looks at the performance of the algorithm on the problems expected in practical applications (network sampling, ordinal data, and alternate opinion diffusion processes) assessed through parameter recovery, modeling of observed opinions, and prediction of future opinions. This is intended to provide context to results produced using this method. The Algorithms manuscript assesses the performance of the algorithm in terms of parameter recovery based on hyperparameters and network and dataset conditions in order to improve accessibility of the method for social science and public health researchers.

2. It would also be interesting for the authors to justify the use of GAs with respect to other heuristic optimization techniques in the study of social network opinion models, for example:

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1078.3250&rep=rep1&type=pdf  

• The linked article uses the DeGroot model but does not present a method to estimate the weights. It identifies individuals as following the DeGroot or voter model based on the model with which their behavior is most consistent. Our previous work (Using a novel genetic algorithm to assess peer influence on willingness to use pre-exposure prophylaxis in networks of Black men who have sex with men), which is cited in the manuscript, includes justification of our choice of the DeGroot model. We added information on why existing methods for fitting opinion diffusion models were insufficient and justification of our choice of a genetic algorithm relative to other bio-inspired algorithms to the Introduction section.

3. In the end the study of hyperparameters does not seem so interesting since finally in section 3. Results it is said that the algorithm found the solutions without taking into account the value set to the hyperparameters in 67.7% of the experiments, which suggests that the evolutionary surface for the DeGroot opinion model should not be very complex, is that so? Please see: https://link.springer.com/article/10.1007/s11042-020-10139-6)

• The high percentage is a result of the objective function and the use of ordinal data, producing a parameter space with many solutions that perfectly fit the ordinal data without recovering the parameters. As such, the relationship between the hyperparameters and ability to find a perfect solution is not particularly interesting, though this information and robustness is beneficial to the user. Instead, the interesting piece is how the hyperparameters can be used to identify a perfect solution with better parameter recovery which is the focus of the Discussion section.

4. why haven't you compared GA with other heuristic optimization techniques? For example: Particle Swarm Optimization, Bacterial Foraging, Simulated Annealing, etc. Please see this article although there are many examples in the literature:

https://www.researchgate.net/publication/271832435_PV_parameter_estimation_using_different_evolutionary_algorithms

• We added a discussion of the advantages of a genetic algorithm over other bio-inspired algorithms and justified our choice of a genetic algorithm in the Introduction section.

5. You use parameter recovery, why not use other techniques? Please explain the advantages of your protocol in the DeGroot model or opinion pramatros.

• We are also interested in modeling observed opinions and predicting future opinions, but good parameter recovery results in the other two. Since parameter recovery is the most important and is not measurable in practical applications, selecting hyperparameters that improve parameter recovery is the priority. We clarified this point in the Measures subsubsection.

Reviewer 3 Report

CALIBRATION OF A GENETIC ALGORITHM FOR MODELING OPINION DIFFUSION

Kara Layne Johnson and Nicole Bohme Carnegie

1. Scope

Is the title explicit, attractive and interesting? 

Is the abstract clear and to the point, stressing both the specific application and the generic aspects of the work?

The paper is mainly addressed to specialists that are interested in the identification of DeGroot opinion diffusion models. The abstract underlines this aspect. The title is explicit and very clear.

2. Does the introduction clearly state the application area?

The "Introduction" section states the application area and the objectives of the work:

- The paper presents a GA devoted to modelling opinion diffusion.

- The authors conducted a simulation study investigating hyperparameter values more accessible to applied researchers

3. Is there real evidence of the practical benefits of the technologies/methodologies introduced (e.g., where it was applied and what improvements resulted)?

Does the drawn conclusion state these clearly?

The paper has, obviously, an applicative character. What is not clear is what is the practical objective of the authors. There are two possibilities:

- The paper could guide the readers about how they can implement their GAs to identify specific DeGroot models.

- The paper could give recommendations on how the control parameters and hyperparameters can be set and modified to use the GA already implemented by the authors to solve the reader's specific problem.

In my opinion, the second objective is more realistic, taking into account how the simulations description and the "Discussion" section are written.

4. Are there generic aspects that make the work applicable beyond a narrow range of applications?

Are these clearly, brought out in the paper?

The strategy to adjust the control parameters and hyperparameters within the GA can be useful for many GAs solving other kinds of optimum problems. Unfortunately, this paper presents these aspects subjected to the problem at hand. A more general way to present how to use the control parameters and hyperparameters could enlarge the paper's addressability.

5. Is the paper correct technically?

The paper is technically correct. The author has very good experience in the field of Gas and DeGroot opinion diffusion models.

6. Is there some aspect, either in theory or application, which is new or innovative?

The authors have implemented a GA whose dynamic is adapted using control parameters and hyperparameters. Both aspects are devoted to the problem at hand. So, we can characterize this work as THE APPLICATION OF KNOWN TECHNIQUES FOR A SPECIFIC PROBLEM.

7. Is the paper intelligible but non-trivial to a practising reader in the field of intended application?

If (AND ONLY IF) the authors' target reader is the specialist interested in identifying the DeGroot model, the paper's structure and work presentation are good. The author makes a very good trade-off between some details concerning the implementation and the understandability of their work.

8.  Is the paper intelligible and of some relevance to practising professionals in other fields?

NO.

9. Is the paper easy to read, i.e.,

• Is it to the point?
• Is it grammatically and semantically simple and correct?
• Are the figures, graphs, etc., clear, explicit and properly labelled?
• Are mathematics essential?
• Are the references complete and relatively easy to obtain?
• Is the length appropriate?

The paper is not easy to read and understand because of the following aspects:

- Section 2.1, "Opinion Diffusion Modeling", presents too briefly the model and the optimization problem:

    = The forward and back transformations are not very clear. A short graphical representation would be clearer.

    = The input data of the optimization problem is not presented (shortly).

- Section 2.3, "Algorithm Calibration", doesn't have a procedural description (flowchart or pseudocode).

- The analysis made in the last section needs a lecturer specialist in the problem at hand.

The other aspects are satisfactory.

TACKING INTO ACCOUNT

- THE NARROW ADDRESSABILITY OF THE PAPER

- THE GOOD PAPER'S PRESENTATION,

AND CONSIDERING WHAT IS ALREADY WRITTEN, I AM NOT ASKING THE PAPER's REVISION.

The drawbacks presented at point #9 can be compensated by a supplementary effort of the interested lecturer.

Congratulations on your work!

Author Response

1. Is there real evidence of the practical benefits of the technologies/methodologies introduced (e.g., where it was applied and what improvements resulted)?

Does the drawn conclusion state these clearly?

The paper has, obviously, an applicative character. What is not clear is what is the practical objective of the authors. There are two possibilities:

- The paper could guide the readers about how they can implement their GAs to identify specific DeGroot models.

- The paper could give recommendations on how the control parameters and hyperparameters can be set and modified to use the GA already implemented by the authors to solve the reader's specific problem.

In my opinion, the second objective is more realistic, taking into account how the simulations description and the "Discussion" section are written.

• The second objective is the purpose of the paper. We clarified this point in the Introduction section.

1. Are there generic aspects that make the work applicable beyond a narrow range of applications?

Are these clearly, brought out in the paper?

The strategy to adjust the control parameters and hyperparameters within the GA can be useful for many GAs solving other kinds of optimum problems. Unfortunately, this paper presents these aspects subjected to the problem at hand. A more general way to present how to use the control parameters and hyperparameters could enlarge the paper's addressability.

• While the intended purpose of the paper is fairly narrow in scope, we added to the Discussion section to highlight that this paper can be applied to other parameter estimation problems with large parameter spaces and many perfect solutions.

1. Is the paper intelligible but non-trivial to a practising reader in the field of intended application?

If (AND ONLY IF) the authors' target reader is the specialist interested in identifying the DeGroot model, the paper's structure and work presentation are good. The author makes a very good trade-off between some details concerning the implementation and the understandability of their work.

• Our target reader is the specialist using this method for their applied research, and the changed to the Introduction section should reflect this.

1. Is the paper easy to read, i.e.,

• Is it to the point?
• Is it grammatically and semantically simple and correct?
• Are the figures, graphs, etc., clear, explicit and properly labelled?
• Are mathematics essential?
• Are the references complete and relatively easy to obtain?
• Is the length appropriate?

The paper is not easy to read and understand because of the following aspects:

- Section 2.1, "Opinion Diffusion Modeling", presents too briefly the model and the optimization problem:

    = The forward and back transformations are not very clear. A short graphical representation would be clearer.

    = The input data of the optimization problem is not presented (shortly).

- Section 2.3, "Algorithm Calibration", doesn't have a procedural description (flowchart or pseudocode).

- The analysis made in the last section needs a lecturer specialist in the problem at hand.

The other aspects are satisfactory.

• We incorporated additional details on the structure of the dataset for which this method was developed and tied it to the transformations in the Transformations subsubsection. We also incorporated a graphical representation of the transformation process. We included a flowchart in the Algorithm Calibration subsection to provide a big picture overview of the process. While the details specific to the opinion diffusion application are necessary for the discussion and are intended for an expert in the field, the content added to the Discussion section highlight the aspects of both that section and the Methods and Results sections that are relevant to other applications.

Round 2

Reviewer 1 Report

My comments have been, more or less, addressed. During the proof-reading, please correct the following typo:

In this section, we first, we explain

->

In this section, first, we explain

Reviewer 2 Report

The authors have resolved and revised the five comments made on the first version of the paper, and since the article has been significantly improved I recommend that it be accepted for publication in the journal.