Toward Ultra-Low Latency SSDs: Analyzing the Impact on Data-Intensive Workloads
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe paper provides a comprehensive analysis of ultra-low latency SSDs and their impact on data-intensive workloads. The empirical study covers a range of storage and application performance aspects, shedding light on the potential benefits and challenges associated with these emerging storage technologies. The following feedback is provided to enhance the clarity, coherence, and completeness of the manuscript.
1. The paper lacks a clear delineation of sections in the abstract. Consider restructuring the abstract to succinctly present the background, objectives, methodology, key findings, and implications.
2. The terminology used in the abstract and introduction needs to be more precise. Consider providing concise definitions for terms like RDMA, NVMe, and SCM for readers who may not be familiar with these acronyms.
3. The paper could benefit from additional explanation and clarity in certain sections, such as the distinction between ultra-low latency (ULL) SSDs and conventional SSDs.
4. The methodology for benchmarking ULL SSDs could be detailed further. Include information on the specific workloads used, the hardware configurations, and any relevant parameters. Provide a clear description of the criteria used to select the examined SSDs and the rationale behind choosing them for the study.
5. Are the 'mixed use' used in this article synthetic workloads? Why should use the mixed use?
6. For these workloads in Section 3, more detailed explanations are needed. Are they representative? Can these workloads be made public?
Comments on the Quality of English LanguageGood
Author Response
I want to convey my appreciation for the comprehensive review of my paper, and your constructive feedback provided has significantly enhanced the quality of my paper.
[Comment 1] The paper lacks a clear delineation of sections in the abstract. Consider restructuring the abstract to succinctly present the background, objectives, methodology, key findings, and implications.
Response: Thank you for your comprehensive review. I have revised the majority of the abstract to ensure it not only summarizes my research findings but also offers a clear delineation of its background, objectives, methodology, key findings, and implications. Your insights are greatly appreciated.
[Comment 2] The terminology used in the abstract and introduction needs to be more precise. Consider providing concise definitions for terms like RDMA, NVMe, and SCM for readers who may not be familiar with these acronyms.
Response: In response to your suggestion, concise definitions for terms such as RDMA, NVMe, and SCM have been added to both the introduction and discussion sections of the revised manuscript.
[Comment 3] The paper could benefit from additional explanation and clarity in certain sections, such as the distinction between ultra-low latency (ULL) SSDs and conventional SSDs.
Response: In response to your suggestion, I have revisited and revised Section 2.1 to provide additional explanations and enhance clarity, particularly focusing on clearly distinguishing between Ultra-Low Latency (ULL) SSDs and conventional SSDs.
[Comment 4] The methodology for benchmarking ULL SSDs could be detailed further. Include information on the specific workloads used, the hardware configurations, and any relevant parameters. Provide a clear description of the criteria used to select the examined SSDs and the rationale behind choosing them for the study.
Response: In response to the valuable feedback, I have included detailed workload information in Section 3.3, shedding light on the specific workloads utilized. Additionally, I have introduced new content to Section 3.1, titled "Experimental Setup," providing comprehensive details about the hardware configurations and relevant parameters. This section now thoroughly explains the criteria employed in selecting the examined SSDs, elucidating the rationale behind my choices for this study.
[Comment 5] Are the 'mixed use' used in this article synthetic workloads? Why should use the mixed use?
Response: In response to your question, I have added clarification in the manuscript that the 'mixed use' scenarios used in this article are synthetic workloads generated using the FIO tool. These synthetic workloads are designed to simulate real-world scenarios and provide a controlled environment for performance evaluation. Furthermore, additional details and answers related to the use of mixed-use scenarios are now available in Section 3.2.1 of the manuscript.
[Comment 6] For these workloads in Section 3, more detailed explanations are needed. Are they representative? Can these workloads be made public?
Response: I have addressed your suggestion by adding concise details in Section 3.3, explaining my choices based on both public availability and widespread industry usage.
Reviewer 2 Report
Comments and Suggestions for AuthorsThis paper compares emerging ultra-low latency SSDs with traditional SSDs. The author's literature review of the research questions and comparative research implications is insufficient.
1. The methods section lacks detailed information about the experimental setup, data sets used, and evaluation metrics. The authors need more information about testing emerging ultra-low latency SSDs for read and write speeds, random and sequential access performance, and performance under specific workload conditions.
2. Lack of hardware and software environments required to prepare for testing, including emerging ultra-low latency SSDs and the latest traditional SSDs for testing. Make sure the two are in similar environments to ensure the comparison is accurate.
3. There is a lack of evaluation comparing emerging ultra-low latency SSDs with the latest traditional SSDs in actual application scenarios.
4. The presentation of results in the Discussion section is unclear and lacks figures. The results should not be simply presented. Charts and tables should be used to clearly demonstrate comparison results.
Author Response
I want to convey my appreciation for the comprehensive review of my paper, and your constructive feedback provided has significantly enhanced the quality of my paper.
[Comment 1] The methods section lacks detailed information about the experimental setup, data sets used, and evaluation metrics. The authors need more information about testing emerging ultra-low latency SSDs for read and write speeds, random and sequential access performance, and performance under specific workload conditions.
Response: Taking into account the insightful feedback received, I've updated Section 3.3 to include detailed information on the specific workloads used. In Section 3.1, titled "Experimental Setup," I've added new content providing comprehensive details about hardware configurations and relevant parameters. Additionally, you can find the detailed performance specifications of all SSDs, including a comparison with legacy SSDs, in Table 1.
[Comment 2] Lack of hardware and software environments required to prepare for testing, including emerging ultra-low latency SSDs and the latest traditional SSDs for testing. Make sure the two are in similar environments to ensure the comparison is accurate.
Response: Thank you for highlighting this concern. I want to clarify that in Section 3.1, I utilized the same system for both ultra-low latency SSDs and legacy SSDs. This approach ensures that the comparison is conducted in similar environments, addressing the need for accuracy in the testing and comparison process.
[Comment 3] There is a lack of evaluation comparing emerging ultra-low latency SSDs with the latest traditional SSDs in actual application scenarios.
Response: In response to your concern, I'd like to assure you that the specific workload details are updated in Section 3.3. I have carefully chosen a comprehensive set of widely recognized and open-source tools, each serving a specific purpose in testing my system across diverse scenarios. For example, PostgreSQL (PgSQL) is used for SQL-related assessments, while in the dynamic realm of NoSQL databases, I employ Aerospike, RocksDB, MongoDB, and FlashX. Ceph is utilized for distributed storage, and the Write Stress Test (WPT) tool is applied for stress-testing under high-stress write scenarios. Other tools like RocksDB, Memcached, Fatcache, Bcache, and PMbench contribute to my assessment in various practical scenarios.
[Comment 4] The presentation of results in the Discussion section is unclear and lacks figures. The results should not be simply presented. Charts and tables should be used to clearly demonstrate comparison results.
Response: I truly appreciate your emphasis on clarity with figures. It's important to convey that my intention is to provide a comprehensive evaluation of various application scenarios, which may contribute to the perceived complexity. In my view, the current charts and tables effectively capture the nuanced comparisons across different scenarios. Your insights are incredibly valuable, and I will certainly keep them in mind as I move forward. Your guidance is appreciated.
Reviewer 3 Report
Comments and Suggestions for AuthorsThe paper presents an empirical study comparing Ultra-Low Latency (ULL) Solid-State Drives (SSDs) with conventional NAND solutions in hyper-scale computing environments. It explores the performance of ULL SSDs under various data-intensive workloads and compares them with the latest conventional SSDs. ULL SSDs' low latencies, high endurance, and good performance under different workloads conclude their suitability for various applications in hyper-scale computing, emphasizing their potential as data storages and memory extenders.
The manuscript is clear and well-written. The results presented in this work are very useful for groups and companies to build and upgrade their servers and data centers. I have no major issues with the paper, only a few minor items described below, and once these are addressed I fully recommend the paper for publication.
Minor Issues:
1. Page 3, line 76-77, “One interesting finding …”. This sentence seems incomplete.
2. Section 3.2.3 and 3.2.4. When testing the performance of ULL SSDs as Cache and Swap devices, it would be more helpful to compare them with the direct using of DRAMs (DDR4 or even DDR5) maybe in a future work. DRAMs are much cheaper these days and the latency and endurance should be much better.
3. What is the performance of these SSDs under a long-time working? Since the active powers of these SSDs are pretty high, should any additional accessories be used to control the temperatures under a long-time working? Continuous full-disk reads and writes tests may be added in a future work.
Author Response
I want to convey my appreciation for the comprehensive review of my paper, and your constructive feedback provided has significantly enhanced the quality of my paper.
[Comment 1] Page 3, line 76-77, “One interesting finding …”. This sentence seems incomplete.
Response: Thank you for bringing the omission in the paper to my attention. The missing sentences have been incorporated into the revised version, and I highlighted them in blue.
[Comment 2] Section 3.3.3 and 3.3.4. When testing the performance of ULL SSDs as Cache and Swap devices, it would be more helpful to compare them with the direct using of DRAMs (DDR4 or even DDR5) maybe in a future work. DRAMs are much cheaper these days and the latency and endurance should be much better.
Response: Thank you for your valuable comment regarding the comparison with DRAM. One main reason I did not compare with DRAM is the challenge in deployment due to capacity limitations. Unlike ULL SSDs, which offer capacities nearing 1TB, the practical implementation of a system with a 1TB DRAM is indeed quite challenging. I understand DRAM costs and capacities might change in the future. Your suggestion is appreciated, and I plan to explore a DRAM comparison in future work, especially if technology advancements make higher-capacity DRAM solutions more doable.
[Comment 3] What is the performance of these SSDs under a long-time working? Since the active powers of these SSDs are pretty high, should any additional accessories be used to control the temperatures under a long-time working? Continuous full-disk reads and writes tests may be added in a future work.
Response: Thank you for your thorough review. Regarding long-term performance, please note that I have already included the sustained read/write performance metrics of ULL SSDs in Table 1. These metrics were chosen to offer a comprehensive insight into the long-term behavior of ULL SSDs. Additionally, in terms of power, the ULL SSDs in my study are commercial products manufactured with robustness and reliability in mind. Therefore, I believe that these SSDs are as safe as other conventional SSDs for similar applications. Your consideration of these aspects is appreciated. If future work involves continuous full-disk reads and writes tests, I will certainly take your suggestion into account.
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsMy previous comments have been mostly addressed. Thanks for the revision.
Comments on the Quality of English LanguageGood
Reviewer 2 Report
Comments and Suggestions for AuthorsAll my comments have been responded to and corrected by the author. I agree that the paper can be accepted for publication.
Comments on the Quality of English LanguageIt is recommended to check English grammar and syntax.
