HyperShield: An Automated Evaluation Platform for Security and Performance Trade-Offs in Virtual Systems^†

Round 1

Reviewer 1 Report

The manuscript is well written, technically sound, and clearly structured. The proposed HyperShield framework addresses an important gap in the systematic evaluation of security mechanisms in virtualized and containerized environments. The experimental methodology is comprehensive, and the results are convincingly analyzed and well contextualized within existing literature. I do not have any major concerns or required changes.

This work provides a valuable and timely contribution to the field of cloud and virtualization security. The open-source nature of HyperShield, combined with its support for both user-space and kernel-space security modules, enhances the reproducibility and practical relevance of the research. The performance evaluation is thorough, and the discussion of trade-offs between VMs and containers is insightful. Overall, the manuscript meets the journal’s standards for quality, clarity, and impact.

Author Response

We would like to thank the reviewer for their time, feedback, and words of encouragement.

Reviewer 2 Report

This manuscript makes a significant contribution by moving beyond the high-level throughput metrics found in existing literature. By developing HyperShield, the authors provide a valuable testbed for reproducible research.

The manuscript primary strength lies in its depth; it systematically analyzes micro-architectural metrics to empirically demonstrate the fundamental trade-offs between performance and isolation.

The finding that the placement of security modules (user space vs. kernel space) can inverse the performance advantage between VMs and containers offers actionable insights for cloud architects designing latency-sensitive systems.

While HyperShield and the micro-architectural data present a strong foundation, the manuscript requires a Major Revision. Strengthening the causal analysis of performance bottlenecks, aligning the title with the research scope, and expanding on the practical implications for large-scale deployments are essential steps to ensure the paper meets the rigorous standards of the journal.

1. Critique on Scope and Title

A primary concern regarding the manuscript is the discrepancy between its title and its actual content. The title, "Systematic Evaluation of Isolation Guarantees," sets an expectation that the study will quantitatively assess the efficacy of security isolation mechanisms against actual threats. However, the methodology and results are almost exclusively focused on measuring the performance overhead incurred by enabling these security modules. To align the manuscript with its findings, the authors must either revise the title to reflect the focus on performance costs or introduce new experiments that specifically measure the robustness of the isolation itself.

2. Analysis of Performance Bottlenecks (Section 6.2)

The interpretation of the experimental data requires refinement, particularly regarding CPU utilization metrics in Section 6.2. The authors currently present low CPU usage on the client side as a possible sign of system availability, but a more critical analysis suggests that this phenomenon actually indicates a system bottleneck located at the security module. Since the security module acts as a single point of entry, its saturation likely creates backpressure that throttles the client into an idle state. The analysis should be rewritten to explicitly identify this throttling effect using system-time metrics from the security node itself to prove that the observed client behavior is a symptom of resource starvation.

3. Evaluation of VM Throughput Discrepancies

The substantial performance gap observed between Docker (9.38 Gbps) and VMs (1.98 Gbps) requires a deeper technical explanation. Given that the experiments were conducted within a single host using internal communication bridges, the network bandwidth should theoretically exceed the physical NIC's 1 Gbps limit. The suspiciously low throughput for VMs suggests that the workload is heavily CPU-bound due to excessive virtualization overheads, including high costs of VM exits and context switching, rather than being network-bound. The discussion should rigorously investigate these specific bottlenecks to explain why the internal communication was significantly degraded in the VM environment.

4. Scalability and Architectural Implications

While the single-node experimental setup provides a controlled environment, the manuscript would benefit significantly from a discussion on how these findings extrapolate to multi-node cloud architectures. The latency and processing overheads measured in this study would likely accumulate in a multi-hop environment where traffic traverses multiple security nodes, potentially rendering certain architectures unviable for real-time applications. Adding a discussion on these scalability implications and architectural recommendations would significantly enhance the practical relevance of the study for large-scale deployments.

Author Response

We thank the reviewer for their valuable insights and suggestions.

[Comments 1:] [ Critique on Scope and Title. A primary concern regarding the manuscript is the discrepancy between its title and its actual content. The title, "Systematic Evaluation of Isolation Guarantees," sets an expectation that the study will quantitatively assess the efficacy of security isolation mechanisms against actual threats. However, the methodology and results are almost exclusively focused on measuring the performance overhead incurred by enabling these security modules. To align the manuscript with its findings, the authors must either revise the title to reflect the focus on performance costs or introduce new experiments that specifically measure the robustness of the isolation itself.]

[Response 1:] [We have changed the title to "HyperShield: An Automated Evaluation Platform for Security and Performance Trade-offs in Virtual Systems" to better align with the manuscript findings.]

[Comments 2:] [Analysis of Performance Bottlenecks (Section 6.2) The interpretation of the experimental data requires refinement, particularly regarding CPU utilization metrics in Section 6.2. The authors currently present low CPU usage on the client side as a possible sign of system availability, but a more critical analysis suggests that this phenomenon actually indicates a system bottleneck located at the security module. Since the security module acts as a single point of entry, its saturation likely creates backpressure that throttles the client into an idle state. The analysis should be rewritten to explicitly identify this throttling effect using system-time metrics from the security node itself to prove that the observed client behavior is a symptom of resource starvation.]

[Response 2:] [While the reviewer's insight about the back pressure and security module being a potential choke point is valid. Our characterization results are from the security module itself. The low CPU utilization is primarily because packets are network and IO-bound rather than CPU-bound.  It also depends on many other factors described in the comprehensive evaluation in section 6. We have highlighted it in sections 5.1 and 6. ]

[Comments 3:] [Evaluation of VM Throughput Discrepancies The substantial performance gap observed between Docker (9.38 Gbps) and VMs (1.98 Gbps) requires a deeper technical explanation. Given that the experiments were conducted within a single host using internal communication bridges, the network bandwidth should theoretically exceed the physical NIC's 1 Gbps limit. The suspiciously low throughput for VMs suggests that the workload is heavily CPU-bound due to excessive virtualization overheads, including high costs of VM exits and context switching, rather than being network-bound. The discussion should rigorously investigate these specific bottlenecks to explain why the internal communication was significantly degraded in the VM environment.]

[Response 3:] [We have improved the depth of our discussion in section 6.1 to describe the virtualization overhead that resulted in lower VM throughput.]

[Comments 4:] [ Scalability and Architectural Implications: While the single-node experimental setup provides a controlled environment, the manuscript would benefit significantly from a discussion on how these findings extrapolate to multi-node cloud architectures. The latency and processing overheads measured in this study would likely accumulate in a multi-hop environment where traffic traverses multiple security nodes, potentially rendering certain architectures unviable for real-time applications. Adding a discussion on these scalability implications and architectural recommendations would significantly enhance the practical relevance of the study for large-scale deployments.]

[Response 4:] [We have added section 7.1. Architectural Implications for Multi-Node Scalability to describe the scalability of HyperShield in a multi-node architecture. We also added section  7.2. Future Research Directions, where we described HyperShield, can be extended for future research to support a multi-node system.]

Reviewer 3 Report

HyperShield stands out by providing a unified, extensible, and open-source framework capable of evaluating security modules deployed in both user space and kernel space. The proposed approach is innovative, as it is not limited to synthetic benchmarks or to simple security mechanisms (e.g., firewalls).

The article represents a solid and relevant contribution to the field of security in virtualized systems. It addresses a highly relevant issue in cybersecurity and cloud computing, namely the evaluation of the security–performance trade-off in virtualized environments (VMs versus containers). The main contribution, HyperShield, stands out by providing a unified, extensible, and open-source framework capable of evaluating security modules deployed in both user space and kernel space.

The paper presents a systematic comparative evaluation of virtual machines versus Docker, integrates microarchitectural metrics (such as TLB behavior, cache misses, IPC, and CPU migration), and analyzes the impact of security module placement (user space versus kernel space). The methodology employed is rigorous and well documented; the experimental setup is clearly described (hardware, software, and established benchmark tools), and multiple configurations are analyzed to enable fair comparisons. The validity of the results is further strengthened by the use of a real-system experimental environment.

The results are well presented, supported by clear and informative visual representations, and correctly interpreted. The authors demonstrate that containers are not always superior to virtual machines, particularly for kernel-space and I/O-intensive workloads.

Weaknesses

1. Only a limited set of security modules is evaluated, which constrains the scope of applicability of the framework.
2. The study does not consider energy consumption or power efficiency, which is increasingly relevant in cloud and edge environments.

Recommendations

To further strengthen the contribution, it is suggested to incorporate an evaluation of energy consumption, enabling a more comprehensive assessment of the efficiency and sustainability of the proposed framework. Additionally, as a direction for future work, extending the experimental evaluation to include a broader range of security modules (e.g., Snort for direct comparison with Suricata) would enhance the robustness and general applicability of the results.

Author Response

We thank the reviewer for their valuable insights and suggestions.

[Comment 1: ] [as a direction for future work, extending the experimental evaluation to include a broader range of security modules (e.g., Snort for direct comparison with Suricata) would enhance the robustness and general applicability of the results.]

[Response 1: ] [We have added section 7.2. Future Research Directions to describe how the HyperShield framework can be extended.]

[Comment 2: ] [The study does not consider energy consumption or power efficiency, which is increasingly relevant in cloud and edge environments.]

[Response 2: ] [We have added section 6.7. Energy and Power Overheads that contains measurement of energy and power consumption.]

Round 2

Reviewer 2 Report

The authors have significantly improved the manuscript by realigning its scope with the experimental findings, transforming it from a specific study on isolation into a comprehensive platform for evaluating security and performance trade-offs.

The revised manuscript offers much-needed clarity on the system-level impacts of security modules, providing a well-supported explanation for performance variations across different virtualization environments. By expanding the discussion to include distributed cloud architectures and multi-hop scenarios, the authors have increased the practical relevance of their findings for large-scale deployments.