Review Reports

Round 1

Reviewer 1 Report

The paper presents a model and technique for naturalistic programming: yet, it is really verbose, with plenty of details in the text that should better be placed in an appendix, failing to guide the reader straight to the key points. The first 6 pages could be dried up to 1.5-2

Intro: in lines 28-30, MDA/MDE should probably be mentioned too; in lines 42-46, discourse is not satisfying, remains too vague - the reader comes across the mixing concept but no motivation is given for mentioning it at that point. in line 78, "refer to the instructions in the code" is obscure, authors should not take anything for granted. More generally speaking, authors should 1) present AOP and explain what it is, 2) put in context the terms used, 3) motivate each claim or provide suitable references. In line 93-95 discourse in quite obscure, authors seem to speak for themselves rather than clarifying issues to the readers.

Background: somehow wasteful, interrupts the flow of discourse, starting from Adam & Eve (112-123), with many claims without references: in order to be effective, claims should be strongly connected to specific cases, not merely rephrasing intro. I'd rather suggest a tailored "related" section at the and of the paper. In lines 175, "limited" support means quite little - please be specific and put in context. Just after, "the last operation" is merely put there without going in depth. My point is that such a background section is only understandable if you are already familiar with the field, i.e. precisely in the case when you do not need to read a background section; otherwise, it is unlikely that the reader understands clearly the point. In section 2.2, "anaphoric", "formalistic", "naturalistic" should better be explained. The last paragraph, lines 197-209, remains quite at a descriptive level, never goes in depth: it is too vague to he helpful.

Section 3: "conceptual model" as a title would be ok, but in the intro it is called "proposed model".. different namings do not help. In lines 216-220, please add an example. In lines 221-226, please focus on explaining rather than glossing. In lines 232, please avoid trivialities such as "deeper analysis"; in the next lines, please dry out the discourse. Moreover, please do not have terms appearing without being put in context first: e.g. in line 249, what do java event have to do with the discourse at hand? A few lines after, advices and join points seem actually to refer to AspectJ, but the discourse is quite confused. Then I have a major concern about the example at lines 261-277: first, the presented example is not the only possible way of expressing the concept in Java, while line 261 seems to suggest that this is the case. Moreover the example "as is" is incomprehensible - no qualifiers, no costructor, println to signal anomalies instead of events.. Apart from that, the formatting is poor (also applies to next example). My point is that you cannot just put an example in the text, leaving the burden of interpreting it to the reader - this is supposed to be the authors's task (btw, in 52 pages, authors could not find a couple of lines to present the AspectJ syntax). The claim in line 296 is highly questionable, one can well do that in Java too. The points at lines 311-320 are confused, the subsequent explanation (lines 321-324) is little helpful either. Lines 325-327 is the perfect example of the poor structuring of this paper: after spending a lot of time on obvious things, authors fly over most critical aspects, that should have deserved more attention and space. Finally, the discourse at lines 334-338 is verbose, then (339-341) authors make no real effort to make their points understandable

Section 3.1 on page 8 could well have been the real start of section 3 (apart from lines 357-359, providing the definition of noun, that should be anticipated). Please add proper references when needed, e.g. significant (371), and avoid trivial statements (374-375, 383-385, etc). In line 386, English is probably not the only language satisfying the point: however, the first paragraph is obvious, the last one (391-392) instead would deserve more space -instead, it is flew over.

In section 3.3, lines 399-410, the long discourse does not highlight the key point: how do you handle that? Lines 411-413 useless.

Section 3.4 finally introduces the notion of circumstance, which however was used before (confusing the reader): lines 416-428 are very verbose, the only relevant point is hidden in lines 435-437. Please remind that when writing papers "Less is more": the more you write, the higher the chances that key aspects are ignored and lost somewhere in the sea of words. In short, the only key sentence is at lines 458-459: the whole previous page could be cut out with little or no loss of information. Similar considerations apply to the next sections too

Section 3.5: please note, you are not explaining anything, just speaking over things that you will actually present *later* in section 3.6 -- this is not good structuring.

Section 3.10.1: since this is not obvious, should be introduced and given some extra words,

Section 3.10.2: too long, readers get lost; lines 581-583, does not explain really..

Section 3.10.3: line 587: I totally disagree, interfaces can be well exploited for the purpose.

Overall, 10 pages of model are really a lot: my suggestion is that this part is strongly summarised, giving the highlights and key choices, and possibly moving details to an ad-hoc appendix. It is not viable to get to page 14 to see something more concrete. Such 14 pages could well be shrinked to 4-5 at most.

Section 4 is quite confused, introduces several concepts and names without putting them in clear relation -- prototype, implementation, language are all mixed up. In such a section I would expect the model of the mapping, i.e. the inspiring principles behind the mapping, not a by-example presentation. Lines 620-626 turn out to be incomprehensible, and the subsequent example has no correspondence. Then, suddenly, section 4.1 starts providing grammars -- yet with no top-level. This is all very confused: symbols like "abstraction_body" are put there with no apparent definition, then (page 15) definitions are given step-by-step instead then giving the whole picture. This is even more confusing (by the way, the rule for a_an is missing, although easy to guess). Going further, the example at lines 680-682 does not illustrate what the text has just presented.

The example in lines 684 uses attribute and real number (never introduced before), the next printout (689-690) is not clearly related so something.. very confused.

Some lines after, lines 713-718: it makes little sense to me to provide just "some" grammar rules. If you mean to be formal, then you need to be formal, and leave no pieces aside. Maybe you could put the whole grammar in a side box and properly comment on it in the main text.

It would also help if examples used rules and items you already presented, instead then asking reader to support forward references --see eg "with John" at line 725. Overall confused.

Lines 763-788, again, please do not introduce grammar rules piecemeal, and please when you do, take care that the examples directly and immediately reflect the grammar rules presented, otherwise the example becomes an obstacle rather than a hint. Sorry to say, but it this page authors seem to speak for themselves following their own thoughts.

I could continue with similar remarks throughout this section: overall, the section is a 20-page long manual -- instead, it should have given the ideas, the main choices, the highlights, the motivations.. forwarding the reader to a suitable appendix or supplementary material for the details, when applicable. In its current form, this is neither acceptable for a journal, nor useful for readers.

Let us skip to Section 5. Al line 1294, Scala should be introduced with suitable references, and acronyms like SBT should be explained for the paper to be self-contained. Again, please explain what you mean by "mixin" (exactly in the Scala sense? however the reader might not be familiar with it), don't be vague. The next page suffers from the same lack of clear structure discussed above: the reader comes across many concepts and names , but these are not presented in a clear order according to a well-defined narration scheme - rather, they are merely put there in the text, letting the reader alone in finding his/her way.

For instance, why mentioning the difference between LL and LR, if you never recall it after? If it is relevant, than it should be explained and used, otherwise just not mention it at all. Generally speaking, the section remains often too vague, too merely descriptive. The example at lines 1321- is unrelated from the previous text, this means generating confusion. The subsequent "explanation" (lines 1330-1346) are only useful to provide a limited general idea, but the reader is unable to actually follow the discourse in a critical way because the discourse remains too generic and "contemplative".

Figure 1 is of little help and illustrates the obvious: the process should instead be discussed in the text.

Section 6: examples use never-defined keywords, like "leftover" and "adjective" (yes I could make a guess, but in a formal paper in a journal I should never be asked to make my own guesses).

The subsequent "explanation" does not explain a lot, since it just rephrases the code. At the same time, it does not explain what it should, i.e. the relationship between the abstract class and the trait it uses. Again, the reader is abandoned and left alone in guessing how these things should stay together. Also, words with a specific meaning (composition) should be used with care, and properly explained.

The syntax at line 1405 was never introduced before; the example used "first", "second", etc that are hardly more understandable than symbols -- (1431) this is actually my main concern with naturalistic programming indeed: symbols have been introduced in the human culture precisely to help referring to things in a precise and direct way, so convincing motivations should be provided to actually show that indirect references are more clear and helpful in practical cases. The reader can well doubt that a longer example quickly becomes incomprehensible because of so many backward references...

Apart from that, the two versions of the example should be put side by side.

Section 6.2: formatting and line spacing should be fixed. With respect to the example at lines 1487-88: it is hard to be believe that so many "these" are clear to a reader's eyes. Where is this actually used? Can you provide more complete and convincing use cases in the real world? I suspect that with 10+ entities, the lack of identifiers can quickly lead to a nearly incomprehensible wording..

Line 1584-5: some Spanish left in the example

Section 7, lines 1630+: please introduce the context somehow better

Lines 1664-end of page: this does not prove that the wording is "natural" as claimed: in fact, most participants seem to have needed time to think about it, and yet many of them failed to interpret the wording correctly.

Apart from that: it is simply unfeasible to go on for seven pages reporting each single test and result - the overall results should be summarised and discussed (the details could be made available as supplementary online material). Instead, discussion and validation are completely missing -- after Tables 1 and 2 on page 44, the section stops at the best part -- not acceptable.

Conclusions are merely a final bla bla, yielding no actual value.

Overall, a potentially-interesting research work sadly drown in a sea of verbosity, that prevents its value to be highlighted. The manuscript should be rewritten according to an effectiveness criterion, taking into account the current drawbacks.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The manuscript presents a highly relevant and interesting body of research, which can have a significant impact on how programming is managed. It can be useful, in particular, to students interested in programming using naturalistic language. In addition, the approach is original and is theoretically and empirically grounded. However, the authors should consider revising some elements of the manuscript that can make it even more appropriate. For example, the problems underlying natural language processing are not limited to ambiguity; other complexities exist. If only ambiguity is to be addressed, this should be made clear; otherwise, other complexities should be considered, even if the authors would like to mention the fact that those are not to be approached in the article. The concept of programming semantics is explained in a rather confusing way, so it would help if the authors could clarify it. The study builds upon a research design that consists of questionnaires, which were conducted using a Likert scale. However, it is not clear why the authors consider a Liker scale of 1-4, rather than 1-5 or 1-7. As far as the organisation of the article is concerned, the article does not follow a logical order of the sections. For example, it is not clear why the related work comes towards the end, and why the methodology just before that. The authors can consider the usual structure: theoretical background/related work, data and methodology, analysis, results and discussion and conclusions/final remarks. As the article is very long, this might help guide the reader. Some references are unclear, so they need to be checked. Also, the referencing system used is not always appropriate, so revising this aspect is crucial. Last, but not least, the text needs a careful revision, ideally by a native speaker of English. As it is, the text is uderstandable, but some of the arguments are not entirely clear. While doing this revision, the authors can also consider revising the typography and punctuation. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

This paper is about SN, a proposal for a general-purpose programming language based on the so-called naturalistic paradigm, born to help overcome the gap between requirements (usually expressed mostly in natural language) and coding. 

The main novelties of the SN design are: (1) having a syntax closer to natural language, but formal enough to avoid ambiguities; (2) being general-purpose and not domain-related; (3) introducing plurals (which can be seen as a way of dealing with data structures); (4) the use of "circumstances" (modelled with aspects) to activate the proper operations among the available ones.

While the proposal seems interesting and with a decise focus on making SN formal, the paper is still far from presenting a proper model for such a language, as a true programming language.

The paper presents the syntax of SN with great accuracy, but no formal semantics is given. Its semantics is presented via discussions based on concepts taken from the natural languages studies and examples, and this is surely insufficient (for instance, the paragraph 196-209 on page 5 is rather obscure). Indeed, without a formalisation, it would not be possible any notion of (partial) correctness or of program equivalence.

Also the compiler, whose formal description could be a way to define SN's formal semantics, is presented poorly and does not give enough details about the language.

Some references about formal semantics are:

https://www.cis.upenn.edu/~bcpierce/tapl/

https://www.amazon.com/Formal-Semantics-Programming-Languages-Winskel/dp/0262731037

https://www.amazon.com/Semantics-Programming-Languages-Structures-Foundations/dp/0262570955

I appreciate that the background of the authors is mainly on natural language, but it would be good at least to tackle this issue in the future work part, with some concrete proposals on how to proceed in this direction.

Another issue is that the compiler is said being based on Scala and AspectJ:

- there is a need of a nutshell introduction to both of them, otherwise the paper is not self-contained;

- very important: you seem to overlook the fact that Scala has a complicated type system and that your translation of (maybe wrong) SN programs in Scala can rise Scala-errors that an SN programmer might not be able to understand and therefore to solve.

- the diagram on page 32 is too high-level to make the reader understand how you combine (waive) Scala bytecode and AspectJ bytecode.

About related and inspirational work: you should have a look to Smalltalk and Smalltalk-related languages and environments. Smalltalk is an object-oriented language, fully formal but with a friendly syntax. See the page of Stephane Ducasse for resources on the subject:  http://stephane.ducasse.free.fr/

Another issue of the paper is that all terminology about natural language is largely left undefined or defined too late in the paper. Terms such that "anaphora" should be listed with their meaning in a glossary, because this paper might be of interest for traditional programming language people without that particular background.

The conclusions are not really convincing because they lack of a methodology to summarise the results of the experiments. In the end, despite the difficulties encountered in making your language used by your students, why do you think it is worth continuing working on SN? I personally believe it is: as said before, trying to bridge the gap between requirements and implementation is a worthy line of research.

In the following, I will make some punctual comments.

. Page 3, line 95: what do you mean with "types" here? Later on it seems you call "type" what in some object-oriented languages is called "class". It would be good to have a small dictionary SN <--> object-oriented languages to recognise the concepts. Another example is "verb": is it similar to "method"?

. Page 4, line 149: in my experience, functional programming is closer to natural language than any other paradigm, as it allows the programmer to express directly inductive definitions.

. Page 6, lines 262-271: the example is quite a bad example of programming, as it does output and computation in the same method. It would be good to see the advantage of writing SN code instead of Java code with good Java code, otherwise it is too easy, SN wins!

. Page 10: I found very interesting your treatment of adjectives, still it is not clear to me if you identifies types (for example the nominal types of Java) with adjectives (it looks like that). Please elaborate.

. Page 10, lines 458-459: "Circumstance" is a fundamental concept in your paper, as it makes SN different from other naturalistic proposal. Put an explicit definition and some more examples. It might be good to have introduced AspectJ before.

. Page 15, line 670: the reader might not know Scala and the mixin operation.

. Page 21, lines 908-913: not understandable without a formal semantics.

. Section 4.15: what are local identifiers for you and how do you deal with them? I fail to see the ambiguities in the examples "divided/divided". In particular, is the sentence "take A and divides it between B" correct or it has typos, for instance "take --> takes"?

Tiny things:

. Page 7, line 303: beware of the ".

. Page 30, line 1294: what is the reference in parenthesis?

. Page 35, lines 1481-1490: different format.

. In Section 8: do not start a sentence with a [...], it is better to say "The paper [...]" or something similar. Moreover, check on "presents/present" for singular/multiple references.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

I appreciate the extensive editing and restructuring from the previous version. Although some minor doubts remain, the paper now is overall (weakly-)acceptable in its current form.

Please

improve header row of Table 1 (using numbers both in header row and column is misleading) check minor English typos (e.g. the title of Appendix A) add a title for the Bibliography section

 

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

The manuscript is publishable in its current form. However, I recommend that authors do some more text editing as language is not up to a high standard. Among others, the authors say e.g. "The article [8] define the elements they identified for the object model as:"

Reviewer 3 Report

Given the short amount of time that was given to the authors to implement the requested major revisions, I believe they did more than an acceptable job. Therefore I am in favour of accepting the paper.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 3

Reviewer 1 Report

The paper is now fine. Perhaps one could slightly reformulate the sentence at line 308, which might appear somehow misleading -- in fact, Appendix B does not contain only the "remaining" elements, but the whole grammar.