A Survey of the Selenium Ecosystem

Round 1

Reviewer 1 Report

SUMMARY

The authors of the paper present the results of a descriptive survey carried out to analyze the status quo of Selenium ecosystem.

The survey has been executed in 2019 with 72 participants from 24 countries. The results reveal how practitioners use Selenium. In particular, the aspects considered are the following: main features, test development, SUT, test infrastructure, testing frameworks, design patterns, community, and personal experience.

Results indicates that: Selenium tester uses Selenium WebDriver and Jenkins to drive Chrome automatically and assess the functional features of enterprise web applications.

OVERALL EVALUATION

I believe that the reported survey results are pretty obvious (at least for a researcher/practitioner working with the Selenium ecosystem) but the same interesting. The study has been conducted in professional way and the paper is clear, well written and easy to read. I believe that this kind of works can contribute to the Evidence Based SE panorama, so I see this paper in positive way.

There are aspects that could be improved, e.g., I will apply statistical tests to show statistical significance of the obtained results (similarly for example to what was done in the paper: Marco Torchiano, Federico Tomassetti, Filippo Ricca, Alessandro Tiso, Gianna Reggio: Relevance, benefits, and problems of software modelling and model driven techniques - A survey in the Italian industry. J. Syst. Softw. 86(8): 2110-2126 (2013)) and I will reduce the appendix A to the list of questions proposed in the survey; results are already reported in the paper and each table makes the appendix heavy. Also the discussion could be improved. At the moment, it is brief and essential and reports mainly the results without giving a depth qualitative analysis of them. However even if it is not perfect, I believe that this paper dealing with Selenium and analyzing its usage can be useful for researchers and practitioners to clarify the Selenium ecosystem usage and giving some suggestions (for example I didn’t know WebDriverManager that is really useful and interesting). To the best of my knowledge, a similar work is not present in the literature, so it deserves publication.

DETAILED COMMENTS

• I will add the word “survey” in the title. ‘Analysis of the Selenium Ecosystem’ as title seems to me too general
• I will tone down a little the abstract in particular the phrase: “aimed to evaluate how the community uses the Selenium ecosystem in terms of toolbox preferences, test development, system under test, test infrastructure, design patterns, or common problems, among other aspects”. The expectations after reading it are too high, in my opinion. For example from the abstract, it seems that the paper analyses Selenium design patterns. Really, only the PO pattern is considered and few is said about it
• In the background, I will briefly explain what are the same-origin policy issues
• Always in the background, I will explain what Electron is. At the minimum, I will add a phrase similar to: Electron is a new way to build cross-platform desktop applications based on Chromium and Node.js. Then, I will add a reference to it.
• In Section 3, Survey design, you wrote: “This part first surveys the nature of the application tested with Selenium (e.g., web or mobile applications)”. Really, in my understanding Selenium is used only to test web apps (e.g., https://www.pcloudy.com/selenium-for-mobile-app-testing/). If you want to test mobile apps, you have to extend it with Appium. So, I don’t understand very well this question.
• You wrote: “Although Selenium can be used as a standalone framework, it is a common practice to embed a Selenium script inside a unit test”. Are you sure that someone can use Selenium without a unit test infrastructure? Without JUnit or TestNG (considering Java as language) you have not the possibility of adding assertions to test scripts so, in my opinion, the usage of Selenium as a standalone framework is useless for testing purposes.
• “Besides, 20.83% of the respondents declared to test also mobile applications” with Appium?
• Really, I don’t understand very well the question: 3.c. Which type of test(s) do you implement? In my daily practice, Selenium is used exclusively to test the functionalities of the system. For, other kinds of testing are more appropriate other testing frameworks. It will be interesting to understand why the respondents used it for security or accessibility testing, for example.
• In the discussion, you wrote: “The fact that most of the respondents declared not using any reporting tool seems an indicator of a lack of comprehensive reporting tools for Selenium.” Another plausible explanation could be that they are satisfied by the basic reports support provided by Selenium. Maybe, for small projects / testsuites, red/green reports are sufficient. Have you considered it?
• Really, Sikuli is not a testing framework. It can be used also for testing but was born for other more general automation purposes: SikuliX automates anything you see on the screen of your desktop computer running Windows, Mac or some Linux/Unix.
• External validity is the validity of applying the conclusions of a scientific study outside the context of that study. In other words, it is the extent to which the results of a study can be generalized to and across other situations: e.g., people, stimuli, and times. The fact that we don’t know if the results obtained in the surveys are statistical significant poses a generalizability problem.
• A work that could be considered in the bibliography (or at least read) is the following: http://jultika.oulu.fi/files/nbnfioulu-201605251903.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

I have read the paper with great interest, as, despite my focus slightly shifted to testing of the mission-critical systems in the last years, I did number of successful test automation projects employing the Selenium platform, so I can say that I have certain insight into the matter.

Overall, I think the paper has good potential and would like to encourage the authors to work in some suggestions which I provide below. I believe they would help to improve the paper to be a really good study on the topic.

POSITIVE POINTS:

I like the easy-to read introduction section well explaining Selenium family and its individual sub-products.

I also like the survey part related to locator, waiting and assertion strategies. It’s highly relevant to know which patterns (or antipatterns) the current practitioners are using, despite they might be sub-optimal.

The survey is fair and brings interesting empirical facts about the developers’ habits in the Selenium community, despite the fact, the size is a bit limited.

SOME SUGGESTIONS TO IMPROVE:

Related work definitely has to be extended for a journal paper. There is number of related studies in the fields of test automation tat are undoubtedly relevant for the context and shall be mentioned there. To help you in this, I will suggest some works that might be relevant for the study.

First, let’s comment on some previous work discussion test automation issues as you do. It would be good to mention some of them. Some suggestions:

Kasurinen, O. Taipale and K. Smolander, “Software Test Automation in Practice: Empirical Observations,” Advances in Software Engineering 2010. doi:10.1155/2010/620836

M. Rafi et al., “Benefits and limitations of automated software testing: Systematic literature review and practitioner survey,” in 7th International Workshop on Automation of Software Test (AST), Zurich, Switzerland, 2012, pp. 36-42.

Bures, Automated testing in the Czech Republic: The current situation and issues, inProc. 15th Int. Conf. Computer Systems and Technologies, ACM, 2014, pp. 294–301.

Alegroth and R. Feldt, Industrial application of visual GUI testing: Lessons learned, inContinuous Software Engineering(Springer, 2014), pp. 127–140.

Also alternatives to Selenium can be mentioned, including some papers doing the comparison. Some example:

Kaur and G. Gupta, Comparative study of automated testing tools: Selenium, quicktest professional and testcomplete,Int. J. Eng. Res. Appl.3(5) (2013) 1739–1743

Another questions relate to the study design: How have you recruited the participants? What were their expertise level in test automation? In which businesses do they create automated tests for? If you don’t have that information, it would be fair to state it explicitly.

Regarding the issues discussed, I am a bit surprised that Maintainability has share only 45.83%. It’s a key point on which, form an economical viewpoint, test automation projects usually fail. However, it’s  just my feeling about the data and I don’t say they are incorrect.

But, it leads to the next point which is maintainability of the automated tests in general. I think this key aspect deserves much more attention in the paper. It’s a pity that you have not added more questions related to this topic into your survey – and I am not going to suggest extending this information (redoing the survey is not feasible off course). But still, I would suggest extend the paper by more discussion on this topic to give it more credibility. Some literature might help:

Alégroth, Emil, Robert Feldt, and Pirjo Kolström. "Maintenance of automated test suites in industry: An empirical study on Visual GUI Testing." Information and Software Technology 73 (2016): 66-80.

Garousi, Vahid, and Michael Felderer. "Developing, verifying, and maintaining high-quality automated test scripts." IEEE Software 33.3 (2016): 68-75.

Also, back to the related work, there are several previous concepts dealing with this problem that could be commented. Some of them are building upon the Selenium specifically, so they are closely relevant to the study. Some examples:

Carino, S., Andrews, J. H., Goulding, S., Arunthavarajah, P., & Hertyk, J. (2014). BlackHorse: creating smart test cases from brittle recorded tests. Software Quality Journal, 22(2), 293-310.

Bures, M., & Filipsky, M. (2016, September). SmartDriver: Extension of selenium WebDriver to create more efficient automated tests. In 2016 6th International Conference on IT Convergence and Security (ICITCS) (pp. 1-4). IEEE.

Christophe, Laurent, et al. "Prevalence and maintenance of automated functional tests for web applications." 2014 IEEE International Conference on Software Maintenance and Evolution. IEEE, 2014.

Bures, M., Filipsky, M., & Jelinek, I. (2018). Identification of Potential Reusable Subroutines in Recorded Automated Test Scripts. International Journal of Software Engineering and Knowledge Engineering, 28(01), 3-36.

Leotta, Maurizio, et al. "Improving test suites maintainability with the page object pattern: An industrial case study." 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2013.

Also, I would recommend extending the conclusion by more discussion in the following point:

“Maintainability, flakiness, troubleshooting, and observability are still open challenges in the development and operation of end-to-end tests. Design patterns such as the page object model (POM) together with proper wait and location strategies could drastically increase reliability of Selenium tests.”

I definitely agree here, and with a naïve industry approach these issues will stay in place for next few decades. Could you elaborate more, how these design patterns could help? And to add more recent design patterns, for instance screenplay pattern? Check these patterns for an inspiration:  https://testguild.com/automation-testing-design-patterns/

If it starts going to extensive, move it to Discussion section, but it shall be in the paper.

And, as few of us in the research are native speakers, let the manuscript proofread by a native.

I hope that would help.

To sum up, I like the paper and I believe my suggestions could help it to get it to really good shape.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Authors have reflected all the essential points I have suggested in my review; hence I recommend accepting the article for publication after minor revision.

I have two suggestions to work in for a minor revision:

Due to its high relevance, could you elaborate bit more on the maintenance problem of automated tests, specially created by Selenium?

Please consider proofreading of the article by a native speaker with a background in technical research. Subjectively, our experience with Grammarly is that it helps in detecting grammar mistakes, but it does not help in achieving desirable English level for a journal publication. Just a suggestion.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf