SCKM: Symmetric Co-Skew Moment for User Selection in Federated Learning

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper addresses the client selection problem under non-IID data, proposes a symmetric co-skew moment measurement method, and designs corresponding client selection algorithms for different scenarios, aiming to reduce communication cost, accelerate model convergence and improve training accuracy. The topic is in line with mainstream research trends with clear research objectives and complete overall framework, possessing certain theoretical and application value. The authors are recommended to revise the paper according to the following suggestions:

1. Increase the number of references and cite more recent high-quality literatures to enrich the research review.
2. The review of related work is shallow and lacks in-depth comparison. Summarize pros and cons of existing methods and point out research gaps. Clearly highlight the novelty and advantages of this work.
3. State the selection basis of hyperparameters and experimental settings to ensure rationality.
4. Add experiments on more datasets and models to fully verify the algorithm generalization ability.
5. Experimental parameters are given without reasonable explanations. Illustrate parameter setting ideas according to research scenarios. Briefly state the influence of parameter adjustment on experimental results.
6. Experimental analysis relies too much on qualitative description. Collect quantitative indicators including accuracy and convergence speed. Use specific data to reflect the superiority of the proposed method.
7. The limitations of the proposed method are not analyzed objectively. Summarize applicable scope and existing deficiencies based on experiments. Put forward reasonable future research directions to complete the discussion.
8. There is no time and space complexity analysis in the manuscript. Derive algorithm complexity theoretically and compare with mainstream methods. Explain practical application advantages based on complexity results.
9. Most experiments are based on standard datasets without real-scene verification. Build practical federated simulation environment for further tests. Verify the practicability of the algorithm in real edge terminal scenarios.

Author Response

We thank the reviewer for the constructive evaluation and helpful suggestions. We have revised the manuscript accordingly. Detailed responses are provided below.

1. Increase the number of references and cite more recent high-quality literatures to enrich the research review.

Answer:
We thank the reviewer for this suggestion. In the revised manuscript, we substantially expanded the Related Work section by incorporating recent literature on federated client selection, heavy-tailed optimization, robust gradient processing, and moment-aware optimization methods published after 2022. We added several recent references on gradient normalization, robust heavy-tailed optimization, higher-order moment regularization, robust medoid sampling, and decentralized optimization under heterogeneous gradient regimes to better position our work within current developments.

2. The review of related work is shallow and lacks in-depth comparison. Summarize pros and cons of existing methods and point out research gaps. Clearly highlight the novelty and advantages of this work.

Answer:
We thank the reviewer for this valuable suggestion. In the revised manuscript, we significantly strengthened the Related Work discussion by explicitly comparing existing geometric and second-order client-selection criteria with the proposed higher-order moment-aware framework. In particular, we now discuss the limitations of cosine-similarity-, norm-, and distance-based methods under heterogeneous heavy-tailed gradient regimes, where low-order statistics may fail to capture asymmetric cross-client interactions. We further clarified the research gap and highlighted the novelty of SCKM as a symmetric third-order co-moment metric designed to characterize complementary gradient structures under non-IID federated settings.

3. State the selection basis of hyperparameters and experimental settings to ensure rationality.

Answer:
We thank the reviewer for this helpful suggestion. In the revised manuscript, we added a dedicated explanation of the rationale behind the experimental settings and hyperparameter choices. Specifically, we clarify that the shard configurations are used to control the degree of statistical heterogeneity in federated learning: smaller shard numbers correspond to stronger non-IID client distributions, while larger shard numbers represent more balanced local data. We also explain that different client-selection budgets are used to emulate different communication constraints and participation levels in practical FL systems. The gradient-summary ratio is fixed to 0.5% to maintain lightweight communication while preserving sufficient structural information for pairwise client comparison.

4. Add experiments on more datasets and models to fully verify the algorithm generalization ability.

Answer:
We thank the reviewer for this important suggestion. We agree that broader evaluation across more datasets and architectures would further strengthen the empirical validation of the proposed method. In the current work, our experiments focus on CIFAR-10 and CIFAR-100 with VGG-based architectures under controlled non-IID federated simulations. To avoid overgeneralization, we revised the manuscript to clearly state this experimental scope and softened claims that could imply validated generalization beyond the tested settings. We also added broader validation on additional architectures, transformer-based federated workloads, LoRA-style low-rank adaptation settings, and large-scale practical deployments as important future research directions.

5. Experimental parameters are given without reasonable explanations. Illustrate parameter setting ideas according to research scenarios. Briefly state the influence of parameter adjustment on experimental results.

Answer:
We thank the reviewer for pointing this out. In the revised manuscript, we expanded the experimental setup to better explain the parameter-setting logic according to practical federated learning scenarios. We now explicitly discuss how shard numbers, selection budgets, and gradient-summary ratios correspond to different levels of data heterogeneity, communication constraints, and client participation sparsity. We also clarify the influence of parameter changes: smaller shard numbers create stronger non-IID settings and make client selection more challenging, while larger selection budgets generally improve participation diversity and convergence stability at the cost of increased communication overhead. These revisions make the experimental design more transparent and better aligned with practical FL scenarios.

6. Experimental analysis relies too much on qualitative description. Collect quantitative indicators including accuracy and convergence speed. Use specific data to reflect the superiority of the proposed method.

Answer:
We thank the reviewer for this valuable suggestion. In the revised manuscript, we strengthened the quantitative discussion of the experimental results. Instead of relying mainly on qualitative descriptions, we now explicitly discuss final test accuracy, convergence behavior, average accuracy, and standard deviation values reported in the tables and figures. We revised the analysis of SCKM-Select and SCKM-Cluster to connect the numerical results with the observed improvements in convergence speed, accuracy, and stability. The updated discussion highlights that the proposed methods achieve higher average accuracy, faster convergence, and stable optimization behavior under heterogeneous non-IID settings compared with the baselines.

7. The limitations of the proposed method are not analyzed objectively. Summarize applicable scope and existing deficiencies based on experiments. Put forward reasonable future research directions to complete the discussion.

Answer:
We thank the reviewer for this constructive comment. In the revised manuscript, we added a more objective discussion of the applicable scope, current limitations, and future research directions of the proposed framework. We clarify that SCKM is mainly suitable for heterogeneous federated learning settings where higher-order cross-client gradient interactions and heavy-tailed update behavior are significant. We also acknowledge several limitations: the current evaluation is limited to controlled federated simulations using VGG-based image-classification benchmarks; compressed gradient sketches may discard part of the fine-grained gradient structure; and higher-order moment estimation introduces additional pairwise computation compared with lightweight first-order similarity metrics. Based on these limitations, we identify adaptive sketching, broader dataset/model validation, transformer/LoRA-based federated workloads, and real-device edge deployment as important future research directions.

8. There is no time and space complexity analysis in the manuscript. Derive algorithm complexity theoretically and compare with mainstream methods. Explain practical application advantages based on complexity results.

Answer:
We thank the reviewer for this valuable suggestion. In the revised manuscript, we added a dedicated complexity analysis for the proposed SCKM-Select and SCKM-Cluster algorithms.

For SCKM-Select, the dominant cost comes from pairwise SCKM computation among candidate clients. Given (K) clients and sketch dimension (d'), the time complexity is (O(K^{2}d')). If all pairwise scores are stored, the space complexity is (O(K^{2})); however, since the average dissimilarity score can be accumulated online, the auxiliary storage can be reduced to (O(K)).

For SCKM-Cluster, the computation is reduced by first partitioning clients into (\hat{C}^{(t)}) clusters and then performing SCKM-based selection within each cluster. The resulting time complexity is approximately
[
O(Kd')+\sum_{c=1}^{\hat{C}^{(t)}}O(|\mathcal{C}_c|^{2}d'),
]
which is more efficient than full pairwise comparison when the clusters are reasonably balanced.

We also clarified the practical advantage of the proposed framework: the selection stage operates only on compressed gradient sketches rather than full gradients. Therefore, the communication cost is (O(Kd')), where (d'\ll d). This makes SCKM-Cluster more suitable for large client pools and resource-constrained federated learning systems than full-gradient-based or full-pairwise client-selection strategies.

9. Most experiments are based on standard datasets without real-scene verification. Build practical federated simulation environment for further tests. Verify the practicability of the algorithm in real edge terminal scenarios.

Answer:
We thank the reviewer for this important suggestion. We agree that real-world federated deployment and edge-device validation are important directions for fully evaluating practical applicability. The current work focuses on controlled federated simulation benchmarks, which are widely adopted in federated learning research for systematically studying the effects of statistical heterogeneity, communication constraints, and client-selection policies under reproducible conditions.

We additionally note that the proposed framework is designed with deployment efficiency in mind, since client selection relies only on highly compressed gradient summaries rather than full-gradient transmission from all participating clients. This design substantially reduces communication overhead and makes the framework compatible with resource-constrained federated environments.

To clarify the current experimental scope, we revised the manuscript to explicitly identify real-device federated deployment, edge-terminal validation, and large-scale practical federated simulations as important future research directions.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper introduces SCKM (Symmetric Co-Skewness Moment), a third-order informational dissimilarity metric for client selection in federated learning under heterogeneous data distributions. The authors propose two operational modes—SCKM-Select for small client pools and SCKM-Cluster for massive deployments—operating on highly compressed (0.5%) gradient summaries. Empirical evaluations on CIFAR-10/100 with VGG-16 demonstrate consistent improvements over cosine-similarity, loss-sketch, and max-diversity baselines.

Strengths:

1. The motivation for addressing heavy-tailed gradient distributions in non-IID FL is timely and relevant.
2. The feature-ranking study provides empirical justification for SCKM over lower-order statistics.
3. The two-mode design demonstrates practical awareness of deployment constraints.

Weaknesses:

1. The paper argues that heavy-tailed gradients motivate higher-order moments, but does not establish why SCKM is optimal (rather than, e.g., third-order cumulants or alternative skewness measures). The β-CGG model in Appendix 9 suggests SCKM appears naturally, but this connection remains heuristic.
2. Table ?? (page 12) has an unresolved reference. The manuscript appears to have formatting corruption.
3. The "elbow-guided clustering" for determining C^* is described vaguely. Algorithm 2 takes C as input but doesn't specify how C is selected adaptively—yet Figure 5 shows the dynamic evolution of C^*.
4. No comparison with recent moment-aware selection methods. References are somewhat dated—most citations are from before 2022.
5. Only VGG-16 on CIFAR variants. Claims about generalizability to LLMs are unsupported by experiments.

Author Response

We thank the reviewer for these important suggestions. In the following, we address each concern in detail and clarify the corresponding revisions made in the updated manuscript.

1. The paper argues that heavy-tailed gradients motivate higher-order moments, but does not establish why SCKM is optimal (rather than, e.g., third-order cumulants or alternative skewness measures). The β-CGG model in Appendix 9 suggests SCKM appears naturally, but this connection remains heuristic.

Answer:

We agree that the original manuscript did not sufficiently distinguish between empirical effectiveness and theoretical optimality. Our intention was not to claim that SCKM is the uniquely optimal higher-order statistic for client selection, but rather to show that third-order cross-moment information provides a practically effective and statistically interpretable signal under heterogeneous and heavy-tailed gradient regimes.

To clarify this point, we revised Sec. 4.4 and Appendix 9 to explicitly state that the $\beta$-CGG construction is intended as an illustrative statistical interpretation rather than a proof of optimality of SCKM. We additionally discuss that alternative higher-order statistics, including cumulant-based or normalized skewness measures, may also be useful under heavy-tailed settings.

We further softened several statements that could be interpreted as overly strong theoretical claims. In particular, we now describe SCKM as a statistically meaningful and empirically effective third-order descriptor of asymmetric gradient interactions, rather than implying that it is theoretically optimal.

Our primary justification for adopting SCKM remains empirical: across the feature-ranking evaluations in Sec. 4, SCKM consistently demonstrated competitive and frequently top-tier predictive performance relative to the other tested geometric and moment-based metrics. The Appendix 9 analysis is intended to provide intuition for why third-order cross-moment structure becomes informative under asymmetric heavy-tailed coupling, rather than to establish a complete optimality theory.

2. Table ?? (page 12) has an unresolved reference. The manuscript appears to have formatting corruption.

Answer:
We thank the reviewer for identifying this issue. The unresolved table reference and formatting inconsistencies were caused by compilation errors in the previous submission. These issues have been corrected in the revised manuscript.

3. The "elbow-guided clustering" for determining C^* is described vaguely. Algorithm 2 takes C as input but doesn't specify how C is selected adaptively—yet Figure 5 shows the dynamic evolution of C^*.

Answer:

We thank the reviewer for pointing out this inconsistency. In the original version, the notation did not clearly distinguish between the round-wise adaptive cluster cardinality and its temporally averaged value. In the revised manuscript, we now denote the cluster number selected at communication round $t$ by $\hat{C}^{(t)}$, which is dynamically determined by applying the elbow criterion over the feasible range $C \in [C_{\min}, B]$. We further define the reported average cluster cardinality as
[
C^{}=\left\lfloor \frac{1}{T}\sum_{t=1}^{T}\hat{C}^{(t)} \right\rfloor .
]
Accordingly, Algorithm 2 has been revised so that $C$ is no longer treated as a fixed input, and the caption and discussion of Fig.~5 have been updated to clarify that the curves show the round-wise evolution of $\hat{C}^{(t)}$, while the dash-dotted lines indicate the temporally averaged value $C^{}$.

4. No comparison with recent moment-aware selection methods. References are somewhat dated—most citations are from before 2022

Answer:
We thank the reviewer for this suggestion. In the revised manuscript, we substantially expanded the discussion of recent heavy-tailed and moment-aware federated optimization literature, particularly works published after 2024. Specifically, we added several recent references on robust heavy-tailed optimization, gradient normalization, higher-order moment regularization, robust medoid sampling, and decentralized heavy-tailed optimization. We also clarified the positioning of our contribution relative to these recent approaches in the revised Related Work section.

5. Only VGG-16 on CIFAR variants. Claims about generalizability to LLMs are unsupported by experiments.

Answer:
We thank the reviewer for this observation. Our intention was not to claim experimentally validated applicability to LLM-scale federated systems, but rather to discuss possible future extensions beyond the current CIFAR/VGG evaluation setting. To avoid overinterpretation, we revised the corresponding discussion in the conclusion section and clarified that transformer- and LoRA-based federated workloads are mentioned only as potential future research directions. We additionally refined several expressions related to deployment scalability to avoid implying experimentally validated generalization beyond the architectures evaluated in this work.

Author Response File: Author Response.pdf