Evaluation of the Nomological Validity of Cognitive, Emotional, and Behavioral Factors for the Measurement of Developer Experience

Round 1

Reviewer 1 Report

In user experience surveys, it is most likely that the different mental models of developers and users lead to poor ease of use. Therefore, developer evaluation tools have research value for understanding the thinking, emotion and behavior patterns of developers. However, the article needs to be revised as follows to allow researchers and readers to better understand the research process and research results.
1. Provide research hypotheses and understand the causal relationship between various variables in the research.
2. Is the scale used in the study different from the framework proposed by Fagerholm and Münch (2012)? The study only selected preliminary survey questions with a CVR of 1.0 to develop the main questionnaire. The number of questions in each aspect of the final survey should be added to understand the difference between the question items in Table 1, Table 2, and Table 3.
3. In the research procedure of the Delphi investigation, an explanation of the background of experts should be added to understand the degree of professionalism, and the investigation procedure of the group discussion should also be explained.
4. The research mainly investigates whether the S and L companies in South Korea can explain the company attributes and the nature of the basic work content of the investigators. In addition, whether work experience will also affect the investigation, and whether different industries have an impact. Supplement the scope of the investigation to be clear about the scope and limitations of the research.
5. In addition, are the deep learning platforms of the two companies in the survey the same? What are the differences? A description should be added to understand whether there are differences in surveys and willingness to use for different platforms.

Author Response

In user experience surveys, it is most likely that the different mental models of developers and users lead to poor ease of use. Therefore, developer evaluation tools have research value for understanding the thinking, emotion and behavior patterns of developers. However, the article needs to be revised as follows to allow researchers and readers to better understand the research process and research results.

1. Provide research hypotheses and understand the causal relationship between various variables in the research.

Thank you for your thoughtful comments to improve our work. Our work did not examine any influences or effects of independent or exogenous variables on dependent or endogenous variables. Instead, we intended to call for the evaluation of the nomological validity regarding whether three high-order constructs of cognitive, affective, and behavioral constructs are able to consists of the high-order construct of developer experience at statistical significance. Responding to your comments, it should be reasonable to provide the research question instead of hypotheses to be tested. The following sentence is now stated in the main text of the section of ‘3.1. Sub-construct of DX’.

“Inspired by the work of Fagerholm & Münch et al. (2012), the present study ex-plores the research question of “three sub-constructs of cognitive, affective, and conative factor consist of developer experience toward the DL platform”.”

2. Is the scale used in the study different from the framework proposed by Fagerholm and Münch (2012)?

Thank you. In fact, the study by Fagerholm and Münch (2012) is not empirical work but conceptually approached. As can be seen in our manuscript, the scale used in our work was developed mainly from the past literature.

The study only selected preliminary survey questions with a CVR of 1.0 to develop the main questionnaire.

The authors selected the measurement items that have the CVR values greater than 0.99 because the number of panelists participated in Delphi study is four. The guideline for the selection of main questionnaire items was already discussed in main text in the section of ‘3.3.2. Delphi survey’. Thank you for your comment.

The number of questions in each aspect of the final survey should be added to understand the difference between the question items in Table 1, Table 2, and Table 3.

Thank you. Responding to your comments, Table 5 now indicates the original number of pilot items.

3. In the research procedure of the Delphi investigation, an explanation of the background of experts should be added to understand the degree of professionalism, and the investigation procedure of the group discussion should also be explained.

Thank you. The authors have added the professionalism of the four experts participated in the Delphi study to the text in the section of ‘3.2.2. Delphi survey’ as below:

“Of the four experts participated in the Delphi, two respondents were classified as having master’s degree, and the remaining two having doctoral degree. Also, they consist of one four-year university faculty, one principal researcher in research institute, and two project managers with 15 years for industry and practice experience.”

Regarding your comments on the investigation procedure of Delphi study, this work performed one-round Delphi by email. As a matter of fact, it is highly recommended that Delphi should proceed with group discussion in offline or desk setting. Due to the prevalence of Corvid-19, however, this work utilized online-based communication by email. The manuscript now includes the following sentence in the section of ‘3.2.2. Delphi survey’ as below:

“In the present work, one-round Delphi study was conducted by email.”

4. The research mainly investigates whether the S and L companies in South Korea can explain the company attributes and the nature of the basic work content of the investigators.

Thank you. The authors have made additions of the attributes and the nature of basic work for the S and L companies as below:

“In detail, S and L companies are characterized by information technology organization that primarily provide software solution and service integration service. Their business is highly driven by the use of AI-based natural language and learning model, smart factory, blockchain technology, and cloud data analysis.”

In addition, whether work experience will also affect the investigation, and whether different industries have an impact. Supplement the scope of the investigation to be clear about the scope and limitations of the research.

Thank you, the authors addressed your comments in #5.

5. In addition, are the deep learning platforms of the two companies in the survey the same? What are the differences? A description should be added to understand whether there are differences in surveys and willingness to use for different platforms.

Thank you for your comments. The authors admit that several potential confounding factor and profiles the reviewer mentioned might impact the interpretation of our findings in the present work. We addressed your concerns by addition of the following sentence in the section of ‘5. Conclusions’:

“It should be acknowledged that the present work did not consider any potential confounding factors or profiles including the type of deep learning platform respondents use in their companies or level of their work experience into the data analysis. These might impact the interpretation of our findings. Future work should replicate our findings with controlling for confounding variables to provide the robust support for the nomological validity of the measurement model proposed in the present work.”

Reviewer 2 Report

Overall Assessment

The paper is properly structured, easy to read (is properly written considering the academic language) .

Abstract

Please consider the following structure (1 sentence each): Introduction, Research Goals, Research Method, Results, Conclusions, Implications.

Authors should avoid abbreviations in the abstract.

manuscript

Please, contextualize each table within the text. There is no chart corresponding guide in the manuscript.

In order to promote the Journal (Applied Sciences), if possible, the published articles of the journal may be appropriately quoted

Author Response

Overall Assessment

The paper is properly structured, easy to read (is properly written considering the academic language) .

Abstract

Please consider the following structure (1 sentence each): Introduction, Research Goals, Research Method, Results, Conclusions, Implications.

Thank you for your invaluable comments for us. The authors re-wrote the abstract as below:

“Background Developer experience should be considered key factor from the beginning of the use of development platform, but it has not been received much attention in literature. Research Goals The present study aimed to identify and validate the sub-constructs and item measures in the evaluation of developer experience toward the use of a deep-learning platform. Research Methods A Delphi study as well as a series of statistical methodologies including the assessment of data normality, common method bias and exploratory and confirmatory factor analysis were utilized to determine the reliability and validity of measurement model proposed in the present work. Results The results indicate that the measurement model proposed in this work successfully ensures the nomological validity of the three second-order constructs of cognitive, affective, and behavioral component to explain the second-order construct of developer experience at p<0.5 Conclusion The measurement instrument developed from the current work should be used to measure the developer experience during the use of deep learning platform. Implication The results of the current work provide important insights into the academia and practitioners for the understanding of developer experience.”

Authors should avoid abbreviations in the abstract.

Thank you for your comment. Now, abbreviations for ‘DX’ and ‘DL’ in the abstract are not appeared in the abstract

Manuscript

Please, contextualize each table within the text.

Thank you, the manuscript included the following sentence as below:

“As illustrated in Table 1, calculated CVRs for the measurement items of 1, 4, 6, 7, 11 thru 14, 20, and 21 are less than 1.0 and these ten items were discarded for the final analysis. With respect to the ten pilot items for affective factors, calculated CVRs for the items of 1 thru 4, 9, and 10 are less than 1.0 and again, these six measurement items were eliminated and not included in the final analysis (Table 2). Among the calculated CVRs for the measurement items of behavioral factors, the values for the items of 4 and 6 have CVRs less than 1.0 (Table 3). Thus, a total of nine measurement items for behavioral factors are considered for the final analysis.”

Also, the authors elaborated the layout of table 5 and table 6 within the main document appropriately, thank you.

There is no chart corresponding guide in the manuscript.

Thank you, the authors have added the research flow by illustrating the Figure 1 into the chapter of Introduction as below:

Figure 1. Research flow for the present work(In file)

In order to promote the Journal (Applied Sciences), if possible, the published articles of the journal may be appropriately quoted.

Thank you. The authors totally agree with your opinion, in that any manuscripts submitted to the target journal to be published should cite several articles published from the journal. The authors tried to find out to-be-cited articles in your journal thoroughly, but unfortunately, we could not find any proper one matched with the subject, theme, or keyword of our manuscript. The authors regret that we could not quote any articles published in your journal of ‘Applied Science’ in our manuscript.

Author Response File: Author Response.docx