Evaluating the Expressive Range of Super Mario Bros Level Generators

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article's topic and the current version is well formulated in organization, methodology and presentation of the results. However, some minor improvements should be made before publication. Firstly in the introduction, the authors should include a paragraph with a discussion about the new findings of the article and the methodology used, and also I suggest to include a summary of experiments and results of the paper.  Another weak part of the paper is the presence of some typos, for instance, the following:

 

 

line 28: “service” should “services”

 

Line 39: “in the figure” should be “in  figure”

 

Lines 41-44: please write a new paragraph “This document is … , respectively.” ; “follows: section” should be “follows: Section”;

“the section” should be “section”

 

 

Line 110: “5 types” should be “five types”

 

 

Line 112: “has also been” should be “was also” 

 

 

Line 172: “as  are books”? Estrange! Please review and modify appropriately!

 

 

Line 175: “these 4 vectors” should be “these four vectors”

 

 

Line 185: “3 types” should be “three types”

 

 

Line 317:  “figure” should be “Figure”

Comments on the Quality of English Language

The article's topic and the current version is well formulated in organization, methodology and presentation of the results. However, some minor improvements should be made before publication. Firstly in the introduction, the authors should include a paragraph with a discussion about the new findings of the article and the methodology used, and also I suggest to include a summary of experiments and results of the paper.  Another weak part of the paper is the presence of some typos, for instance, the following:

 

 

line 28: “service” should “services”

 

Line 39: “in the figure” should be “in  figure”

 

Lines 41-44: please write a new paragraph “This document is … , respectively.” ; “follows: section” should be “follows: Section”;

“the section” should be “section”

 

 

Line 110: “5 types” should be “five types”

 

 

Line 112: “has also been” should be “was also” 

 

 

Line 172: “as  are books”? Estrange! Please review and modify appropriately!

 

 

Line 175: “these 4 vectors” should be “these four vectors”

 

 

Line 185: “3 types” should be “three types”

 

 

Line 317:  “figure” should be “Figure”

Author Response

Thank you for your thorough review and the provided comments.

Comments 1: the authors should include a paragraph with a discussion about the new findings of the article and the methodology used, and also I suggest to include a summary of experiments and results of the paper.

Response 1: We agree with the comment, we have added the requested paragraph.

Comments 2: presence of some typos

Response 2: We have fixed the typos mentioned, except for the one in line 172. The word that appears in the article is "blocks" not books, and it is the correct word.

 

Reviewer 2 Report

Comments and Suggestions for Authors

In this article, the authors carry out an expressiveness study of open-source automatic level generators for the videogame Super Mario Bros (SMB). These are: (1) Multi-Population Genetic Algorithm; (2) Deep Convolutional Generative Adversarial Network and Covariance Matrix Adaptation Strategy; (3) Markov Chains. The article’s main contribution is comparing three generative spaces using Expressive Range Analysis. Two of the three evaluated implementations are peer- reviewed, the third is authors' implementation of Markov Chains, and finally, these are contrasted with the SMB levels used as training data. The levels have been analyzed under nine metrics through heat charts and box/whisker graphs.

The expressive range analysis is useful for the early evaluation stages, as the heat maps allow us to clearly visualize how the algorithms exhibit uncertain desired characteristics. Genetic algorithms show a wide expressive range despite early convergence. The presented example uses four different populations, allowing high locality in the search space and generating diverse content. Markov Chains are efficient due to their simplicity and the speed with which they are executed. It is important to have a large corpus of levels to guarantee greater diversity in the results. However, like machine learning methods, they are complicated to control. Generative Adversarial Networks produced good content but were sometimes incoherent, not very diverse, and had a limited expressive range.

 

The paper fits well to the Algorithms journal. It is well written and illustrated. However, I would recommend to present a list of multiple abbreviations. The results are of interest for the considered problem domain.

 

I recommend the paper for publication in Algorithms in its present form.

Author Response

Thank you for your review and comments.

Comments 1:  I would recommend to present a list of multiple abbreviations.

Response 1: Page 13 already includes a list of abbreviations.