QuickFaaS: Providing Portability and Interoperability between FaaS Platforms
Round 1
Reviewer 1 Report
In this work, the authors propose a tool to support multi-cloud serverless functions. This tool seems to be functional and enables new deployment models for java based applications in Azure and GCP.
The strength of the work is the extensive number of use cases and experimentation.
However, overall the novelty of this work is lacking. Tools for multi-cloud and multi-provider support for serverless exist already in multiple forms, e.g., https://gitlab.com/josefspillner/lambada. Furthermore, while QuickFaaS is supposed to be extensible, the current type of application is limited to GCP/Azure Compute plus an object store. That's not that impressive considering the number of services that might be used in composition with a FaaS function.
A new multi-cloud tool can be novel as well, but only if you address a new or different issue other than: "can it be done". Overall, the current state of the manuscript lacks scientific merit. What is the purpose of QuickFaaS? Is it to improve development speed? (if so, show that). Is it to develop redundancies, then show that? What are specific issues with redundant deployments? E.g., how would you manage the traffic? Would a user notice the switchover? How did you specifically address this in the design? Moreover, it's nice to have a tool to deploy and develop multi-cloud solutions, but what about the monitoring, and observing of these systems? Based on the experiment section, observability is still done by manually looking at both cloud platforms' dashboards. Will this not create a similar lock-in issue as before? I now need tools and strategies (per cloud provider) to monitor and ensure quality for users.
Furthermore, the related work should be improved significantly, it only takes a few searches to find other work that proposed typical faas use-cases (which could strengthen the experiments, e.g., Eisman), works that show the typical usage of FaaS (informing requirementes for a tool like QickFaaS, e.g., Leitner) and tools that aim to help with migration and or multi-cloud, e.g., Lambada. Currently, the work only has 10 scientific sources in the references, the rest are blogs and websites.
Author Response
First and foremost, thank you for your time to review our submission.
We do agree with you that there are already several other tools that help in managing multi-cloud complexity, such as IaC tools. However, as mentioned in the article, none of them currently support cloud-agnostic FaaS developments to multiple platforms. Also, the Lambada tool referenced in the review uses a FaaSification process to produce serverless functions that are only compatible with AWS. From what we can tell, there's no mention of it being a cloud-agnostic approach to develop FaaS applications. Following the reviewer's suggestions, we introduced several new scientific references in the related work and across the paper.
Our contribution starts with the characterization of functions and the deployment related-entities, using models, independently of the language. The main purpose of QuickFaaS, as specified in the Abstract, is to enable developers to reuse their serverless functions in different cloud providers with no need to change code or to install extra software, tackling one of the software engineering challenges identified by the SPEC group regarding FaaS - «Cloud functions are devoid of operational logic, theoretically allowing the functions to easily be moved to other FaaS platforms. However, currently each cloud vendor requires a different definition of a cloud function.». The goal was to do so minimising the overhead introduced in the deployed code, namely, execution time, memory usage and package size, which was demonstrated in the results. Several clarifications were done in the paper regarding this discussion, namely in the introduction and section 4.3.
Currently, we don't support agnostic monitoring of the deployed serverless functions, but we do agree that this feature should be considered for future releases of the application in order to minimise the need to manually interact with different platforms in a multi-cloud solution. Because the monitoring part is not the focus of QuickFaaS, the results were collected using cloud-specific API and metrics. This was added as future work.
An initial and work-in-progress version of QuickFaaS was presented at the 2022 European Conference On Service-Oriented And Cloud. The initial work had positive feedback and there were some academics that were interested in using the tool in cloud-based courses. Recently, we contacted the team from Pulumi Cloud Framework who also revealed interest in our view, showing this is a topic relevant to the industry.
Reviewer 2 Report
Developers often face cloud-specific requirements when developing FaaS applications. In order to enables them to reuse their serverless functions in different cloud providers environments with no need to change a code or to install extra software, the authors present and propose for use a desktop tool for multi-cloud interoperability (QuickFaaS) that targets the development of cloud-agnostic functions and FaaS implementations. Therefore the proposed cloud-agnostic approach offers developers better portability for FaaS applications and would contribute to the mitigation of vendor lock-in in cloud computing.
The various entities (Authentication Mechanism, which is based on the OAuth 2.0 protocol: Figure 2, FaaS Deployment: Figure 3, Function Definition; Figure 4) that model each the three main modules that compose the cloud-agnostic solution (presented in Figure 1), together with a number of use cases (thumbnail generation, store translation, search blobs) to exemplify the usage of cloud-agnostic functions are presented in Section 3.
The architecture (Figure 8), and the technologies for the proposed desktop tool (QuickFaaS) are given in section 4, together with the structure of the application described by the class diagrams that are designed using UML. The diagram for each of the 3 modules that compose the cloud-agnostic solution is presented and detailed in a separate subsection of section 4.
Section 5 evaluates different metrics to measure the impact of the proposed approach on the function’s performance and the evaluation of the proposed solution is done by measuring the impact of a cloud-agnostic approach on the performance of the function, compared to a cloud-non-agnostic one. The evaluation result indicates that the cloud-agnostic approach does not have a significant impact on the performance of the function, which demonstrates the validity of the proposed solution.
Author Response
Thank you for your time to review our submission.
We also agree that the proposed cloud-agnostic approach offers developers better portability for FaaS applications, contributing to the mitigation of vendor lock-in in cloud computing. The evaluation results in fact indicate that this approach does not have a significant impact on the performance of the function, demonstrating the validity of the proposed solution.
Reviewer 3 Report
This paper aims to introduce a multi-cloud interoperability desktop tool targeting cloud-agnostic functions development and FaaS deployments.
It is an important problem, but it is also commonly overlooked by the developers who are attached to a single cloud provider. So it is interesting that the authors notice this issue, and solve it with a creative idea.
Overall, the paper is well written. The presentation is clear and the evaluations are thorough.
The reviewer has only one concern (question) that is the solution seems to be built based on the Kotlin-Gradle plugin for Java. So how can the solution be implemented for other languages, like Python, etc? If the solution is only available for Java, it will greatly limit the usability and applicability of this work.
Therefore, the reviewer would like to have this problem get addressed before the paper is published.
Author Response
First and foremost, thank you for your time to review our submission.
Regarding the concern about the function implementation language, our contribution focuses on the characterization of functions and the deployment related-entities, using models that are independent from the language where the functions will be implemented. Currently, the implementation of QuickFaaS uses the JVM ecosystem languages, such as Java and Kotlin. However, the article shows through the characterization of the models, and the implementation for JVM, how the same approach would be possible for other languages.
The models are the same for all environments, and the proposed solution with the invocation of a hook function is an easy concept to be implemented in other languages such as C#, JavaScript or Python. The existing programming classes are also prepared to integrate new runtimes with no need to create extra classes. The task that would take more time to be completed would be the development of the cloud-agnostic libraries for that runtime. We have taken into account your comments and addressed this topic in the introduction and in sections 4.3.2 and 4.3.3.
Despite the tool being built using Kotlin-Gradle, it's possible to start new OS processes for the execution of commands that compile code written in other programming languages. Scripting languages such as Python or JavaScript don't need to be compiled, so the required tools will only be used to download missing dependencies, such as NPM for Node.js packages, or PIP for Python packages.
Round 2
Reviewer 1 Report
Firstly in response to the remark on the last review. Lambda was just one example where small changes in the code generator would enable support for other platform deployments. More tools, such as https://spring.io/projects/spring-cloud-function, can also be used to support multi-cloud deployments, including code generation. This is by no means an issue for the content of the work, as automating this process has many facets, e.g., automating code adaptation, deployment, monitoring, roll-outs, testing, etc. Thus, the point in the last review was, that the problem scope in the article could be improved. After rereading it, this is still not clear. Is it just repackaging code automatically and deploying it (after manual setup of each deployment), or is it rewriting code and deploying it fully automatically, providing some credentials? Clearly pointing out what is automated by QuickFaaS and why it merits a different tool for this combination of automated steps would be more convincing. Also, in the introduction 64-66, QuickFaaS is introduced as a tool for agnostic cloud functions. In the conclusion 877-878, it's presented as a programming model and authentication library having this information already in the introduction could manage a reader's expectations. However, that does not mean QuickFaaS is not an interesting and useful tool for industry and science. It is more a question of presentation.
More to the point, the engineering issues discussed in the paper, how to represent hook functions, how to enable authentication, or how to deal with common event sources, could also be archived with other tools such as Lambda, SpingBoot, or Serverless Framework. Given this problem and sufficient time, a developer would come up with a workable solution. What is novel or more important to this work (in my view) is the modelling of FaaS and describing how this model can be used for better cloud-agnostic implementations. However, the current version focuses more on detailed engineering issues than on this model. The paper could be significantly improved by focusing more on the model and its implication, using QuickFaaS's implementation to show how to do that for selected issues. However, this would significantly change the nature of this paper.
Alternatively, leaving the current focus of the paper (on the engineering issues) it would still help to focus more on the issues that arise from adapting the model. For example, in the description of the Authentication mechanism, isn't it normal to start a local server to listen for OAuth responses? Wouldn't this be the same for a VM agnostic deployment tool or any other Non-FaaS tool that wants to use GCP and Azure? Removing or shortening these sections would help the reader find the crucial information.
Overall, a more focused and maybe shortening of some parts of the article would bring to light the novelty of the cloud-agnostic model and quickFaaS as a supporting tool.
Author Response
First and foremost, thank you for your time to review the new version of our paper and for the comments to improve the document. It is our opinion that the document was improved by following your recommendations.
We do agree with your comments and suggestions, in particular, the comment where you point that the paper could benefit from focusing more on the models. For that reason, we now give more relevance to the models, and we included additional information regarding the issues developers can come across when adopting each of the cloud-agnostic models, giving a more adequate view of the methods used in the research. We also took into account other comments, like the ones about automated deployment and the writing of implementation details.
We've included a “diff” PDF file where you can verify the differences from the previous submission more easily.
