FMT-SVM: A Communication-Efficient Federated Multi-Task Support Vector Machine Framework for Healthcare

Abstract

Federated learning has become a promising paradigm in the training of decentralized machine learning models across clients without sharing raw data, thereby preserving privacy. Current federated support vector machine methods are mainly based on the learning of a single global model, which inadequately addresses the challenges presented by heterogeneous and non-IID client data distributions. To overcome these limitations, we propose FMT-SVM, a novel federated multi-task learning framework that jointly trains both binary and multi-class classification tasks within each client, where the client uses a unified convolutional neural network encoder to extract common features among tasks, which are passed to task-specific linear SVM heads dedicated to each classification task. By leveraging a primal optimization integrating task covariance and global consensus regularization, FMT-SVM explicitly models relationships between heterogeneous tasks and enforces alignment across clients, effectively handling the non-IID nature of data distributions. Unlike traditional FL methods that exchange entire model parameters or large support vector sets, our method communicates only the compact SVM heads during aggregation, greatly reducing communication overhead and enhancing scalability for clients with limited bandwidth. To further enhance privacy, Gaussian differential privacy mechanisms are applied to client updates, balancing privacy preservation with predictive performance. Experiments are performed on two medical image datasets: the Pediatric Pneumonia Dataset and the Breast Ultrasound dataset, demonstrating that the FMT-SVM framework achieves competitive accuracy on both binary and multi-class tasks while maintaining communication efficiency and privacy guarantees. These results highlight the capability of the proposed FMT-SVM framework as a practical, scalable, and privacy-aware solution for the federated true multi-task learning problem in sensitive healthcare applications.

1. Introduction

The advent of big data in the healthcare domain has led to exponential data growth that is driven by the widespread adoption of electronic health records (EHRs), wearable devices, genomics, and medical imaging. Algorithms based on artificial intelligence (AI) and machine learning (ML) have enormous potential to produce useful findings for various healthcare-related applications such as disease prediction, early diagnosis, personalized treatment planning, and resource optimization due to the volume, velocity, and variety of such data [1,2]. But these AI- and ML-based applications needs large and diverse datasets to achieve high performance and robust models, which is important for real-world clinical deployment [3]. But in the healthcare domain, this poses privacy and security concerns and may violate data regulations such as the Health Insurance Portability and Accountability Act (HIPAA) [4] in the United States; the General Data Protection Regulation (GDPR) in the European Union; and other regulations that strictly govern the collection, storage, and sharing of personal health information (PHI) [5]. Hence, it is difficult to build robust ML models that require centralized data from multiple healthcare institutions while adhering to these regulations [6].

To address these challenges, Federated Learning (FL) has emerged as a distributed/decentralized paradigm in which multiple data owners collaboratively train an ML model [7,8]. Furthermore, only model parameters are communicated–not sensitive raw data–thereby preserving privacy.

Despite having many advantages, standard FL presents some key technical challenges [9]:

• Data heterogeneity: Clients tend to have non-independent and identically distributed (non-IID) data, which might prevent model convergence and generalization.
• Communication efficiency: There is frequent communication between clients and the central server for exchange of model updates, which results in high communication overhead, limiting applicability in bandwidth-constrained environments [10].
• Privacy leakage risks: Sharing only model updates can unintentionally reveal confidential information [11].

These challenges are more prominent in healthcare applications, where data is mostly heterogeneous, privacy is a major concern, and computational resources may be limited [12]. Deep learning FL-based methods have, however, been used effectively in healthcare applications including medical imaging [13,14,15,16,17,18,19], EHR analysis [20], and prediction of disease [21]. Traditional models such as Support Vector Machines (SVMs), which are strong in theory, are interpretable, and perform well on small datasets, are still a mainstay in clinical ML applications. SVMs are especially appealing in healthcare settings, where transparency and auditability are paramount, due to their interpretability, which is a critical requirement for clinician trust and decision support, and their robustness to small datasets, which is common in rare disease settings or limited-resource environments [22].

However, traditional distributed SVM approaches such as iterative Exchange of Support Vectors [23], incremental learning [24], and Cascade SVMs [25] are often built on the dual formulation, relying on the exchange of intermediate variables such as support vectors (SVs). While communication-efficient, these methods can compromise privacy, as SVs often represent high-influence data points near decision boundaries. Other distributed approaches, such as Distributed Block Minimization [26], ADMM [27], and Distributed Dual Ascent [28], focus on sharing local model parameters rather than SVs, but they are generally limited to linear models and still require the exchange of intermediate solutions.

In federated settings, dual-form SVMs face notable challenges such as potential privacy leakage from SV sharing and high computational costs of kernel operations. Primal-form SVMs mitigate direct SV exposure but are susceptible to weight divergence in non-IID environments, where local models drift and hinder global convergence. These limitations have motivated the development of FL-specific SVM algorithms that explicitly target communication bottlenecks and privacy preservation and preserve model interpretability. Approaches such as cPDS [29] promote sparsity to reduce communication overhead while maintaining transparency, though this method still relies on SV sharing. BDSVM [30] addresses this by using Iteratively Re-Weighted Least Squares (IRWLS) to exchange kernel Gram matrices instead of raw SVs, improving privacy but introducing scalability issues, as Gram matrices grow quadratically with dataset size. Furthermore, Federated Kernel Support Vector Machine (Fed-KSVM) [31] improves scalability by employing random Fourier features and block boosting, enabling efficient primal–dual training with only model parameters exchanged. SecretSVM [32] enhances privacy through secure multi-party computation (SMPC)-based cryptographic protocols and secret sharing, while Support Vector Federation (SVF) [33] applies vector displacement to perturb SVs before transmission, reducing the risk of data leakage. Other methods, such as TurboSVM-FL [34], use SVM on the server side in FL for global aggregation, introducing selective aggregation methods where SVMs are fit on shared client-side latent class embeddings, reducing communication cost and accelerating convergence.

Despite recent advances, most federated SVM studies remain confined to single-task learning, where each predictive model is trained independently for a specific outcome. This overlooks the potential of multi-task federated learning (MTFL) to exploit shared structures among related problems, a common scenario in healthcare, where comorbid conditions often exhibit overlapping risk factors, shared genetic markers, and interconnected behavioral triggers. Similarly, in real-world clinical settings, the joint modeling of such related tasks can improve the predictive performance of the framework by enabling cross-task knowledge transfer. However, traditional SVMs were not designed for federated environments, and extending them to support MTFL introduces several challenges. Furthermore, most existing MTFL methods treat each client’s data as a separate task or focus on neural network-based models, often lacking explicit modeling of multiple distinct tasks within a client [35,36,37]. These include the coordination of distributed optimization across sites, safeguarding of patient privacy while limiting SV exposure, minimization of communication overhead, and assurance of consistency in model predictions.

Several personalized FL methods, such as FedPer [38], FedRep [39], CPFL [40], and FedBABU [41], allow for client-specific heads over shared global encoders, improving performance under non-IID distributions. However, these methods remain inherently limited to a single task per client, lacking mechanisms for joint modeling of multiple tasks within a client. Alternatively, MTFL methods such as MOCHA [35] treat each client as a separate task and optimize task-specific models with inter-task regularization. While theoretically appealing, MOCHA is not built to handle multiple heterogeneous tasks per client, which is a common scenario in healthcare, e.g., simultaneous detection of pneumonia presence and disease subtype classification from chest X-rays. Furthermore, existing deep MTFL approaches [37,42,43] often transmit large model parameters, rely on deep neural network architectures, and provide limited privacy guarantees, restricting their applicability in resource-constrained clinical environments [44].

To address these gaps, we propose a novel multi-task federated SVM framework called Federated Multi-task Support Vector Machine (FMT-SVM) that integrates SVM-based models with both federated and multi-task learning principles. Our approach leverages a primal optimization strategy that efficiently learns across clients while maintaining global consensus without compromising privacy. The proposed architecture uniquely combines an intra-client CNN-based feature extractor with linear SVM heads for each task within a federated multi-task learning (FMTL) framework. Unlike traditional neural classifiers, we introduce a margin-based learning mechanism into a federated deep learning pipeline, which helps in building robust and generalizable decision boundaries across non-IID and imbalanced clients. It also promotes scalability, since additional SVM heads can be added for additional tasks without modifying the underlying encoder.

Contributions

The main contributions of this work are outlined as follows:

1.

We propose FMT-SVM, a primal-only federated multi-task SVM framework that shares only task-specific heads, thereby achieving up to a 490× reduction in communication overhead while preserving privacy.

2.

FMT-SVM incorporates task-aware covariance regularization to capture inter-task relationships, improving generalization across heterogeneous clients.

3.

Each client employs a compact CNN encoder to extract shared feature representations efficiently, a setup that is suitable for edge or resource-constrained healthcare systems.

4.

We perform comprehensive experiments on two real-world multi-task medical datasets that demonstrate FMT-SVM’s superiority over FedAvg and other federated SVM baselines in terms of both accuracy and communication efficiency.

5.

FMT-SVM is a scalable, privacy-preserving, and resource-efficient framework for multi-task healthcare and other related applications.

2. Related Work

SVMs are well-established models known for their robustness and effectiveness in classification tasks [45]. While FL has primarily focused on neural network models [8], integrating SVMs into FL frameworks poses unique challenges, particularly due to the privacy concerns linked to SVs and the computational complexity of kernel methods. This section provides a comprehensive review of prior research at the intersection of FL and SVMs, as well as the development of MTL within federated settings. We organize this discussion into two main threads: (1) the role of SVMs in FL, which we further categorize into what to federate (using SVM as the core predictive model) and how to federate (leveraging SVM principles to enhance aggregation), and (2) existing approaches to MTL in FL.

2.1. SVMs in Federated Learning

The integration of SVMs into FL has been explored in the literature from two complementary perspectives: SVM as the core predictive model in FL and SVM as an enhancement mechanism for global aggregation in FL.

2.1.1. What to Federate: SVMs as Core Predictive Models in FL

Several studies have focused on adapting SVM training for FL, emphasizing privacy preservation and computational efficiency.

Hsu et al. [46] proposed an FL-based framework that integrates an SVM trained via stochastic gradient descent (SGD) with SMPC to collaboratively train malware classifiers across multiple devices without exposing sensitive local data. This approach ensures privacy through additive secret sharing but is limited to linear SVMs and primal optimization, without explicit consideration of kernel methods.

Ref. [31] proposed Fed-KSVM, which addresses the scalability challenges of kernel SVMs by approximating kernel mappings through locally constructed random Fourier features, thereby reducing the problem to a linear SVM. They proposed a novel block-boosting algorithm that partitions the optimization into manageable subproblems within a primal–dual framework. This design enhances memory efficiency and communication privacy by exchanging only model parameters, not raw data or support vectors.

Ref. [33] proposed SVF, which introduces a dual-formulation-inspired method for federated SVM training and avoids sharing exact support vectors by transmitting perturbed versions using vector displacement techniques. This approach preserves privacy while enabling flexible kernel use without the need for explicit feature mappings. However, performance may degrade due to perturbation noise, and scalability to multiple tasks remains unexplored.

Ref. [30] proposed a budget distributed SVM called BDSVM that extends budget SVM principles to federated settings for non-IID data. By employing Iteratively Re-Weighted Least Squares (IRWLS), BDSVM circumvents the need to exchange support vectors by sharing kernel Gram matrices over a common architecture. This approach enhances privacy and robustness but is computationally intensive and also assumes the compatibility of kernel architectures across clients.

Ref. [29] proposed cPDS, a primal–dual distributed optimization method designed specifically for sparse SVM training on federated healthcare datasets. By promoting sparsity, cPDS enhances model interpretability while maintaining computational efficiency but is limited to binary classification problems.

Ref. [32] proposed SecretSVM, based on threshold secret sharing and distributed consensus, to train SVMs securely in federated environments, particularly for IoT and smart-city use cases. Rather than optimizing traditional SVM objectives, this approach frames training as a secure computation protocol, preventing collusion and ensuring strict privacy guarantees.

2.1.2. How to Federate: SVMs as Aggregation Enhancements in FL

In contrast to using SVMs as predictive models, few works have used SVM principles to enhance the efficiency and robustness of the FL aggregation process; such examples include TurboSVM-FL, proposed in [34], which does not train SVMs directly but, rather, applies SVM concepts to enhance aggregation. The client sends its latent class embeddings to the server, where an SVM is fed onto these embeddings. This selective aggregation reduces communication overhead and accelerates convergence, showing how classical ML principles can benefit neural network-based FL systems.

Table 1 summarizes the SVM-based FL methods. These methods present innovative ways in which SVMs can either serve as the core learning model or help in efficient model aggregation within FL frameworks. However, they typically do not address the need for multi-task modeling or inter-task knowledge transfer, both of which are central to many healthcare applications.

Table 1. Comparison of FMT-SVM with existing federated SVM methods.

Method	Multi-Task	Privacy	Optimization	Aggregation	Communication
	Support	Mechanism			Efficiency
Fed-KSVM [31]	No	Parameter Sharing	Primal	FedAvg	Medium
SVF [33]	No	Perturbed Support Vectors	Dual	Vector Displacement	Medium
BDSVM [30]	No	Kernel Gram Exchange	IRWLS	Fixed Budget	Low
SecretSVM [32]	No	Secret Sharing (SMPC)	Secure Computation	N/A	Low
TurboSVM-FL [34]	No	Model-Embedding Aggregation	N/A	SVM-based Selection	High
FMT-SVM (Ours)	Yes	Model Parameter Sharing	Primal	Covariance-aware	High

Note: N/A denotes “Not Applicable” or unavailable results.

2.2. Multi-Task Learning in Federated Learning

Multi-task learning (MTL) aims to improve generalization by learning multiple related tasks simultaneously, allowing models to exploit shared structures and task-specific nuances. In FL, where data remains decentralized across clients, MTL provides a principled way to handle both statistical heterogeneity and task-level diversity, especially in complex real-world domains such as healthcare.

Early federated MTL works, such as that proposed in [47], primarily focused on a general framework for federated MTL under the assumption that each client’s data distribution is a mixture of latent tasks. The proposed EM-style optimization enables personalization, even for unseen clients, and offers convergence guarantees. However, this method is still grounded in neural network models and does not extend to classical ML formulations.

Another line of work including VIRTUAL [36] generalizes federated MTL to non-convex settings using a Bayesian graphical model framework with variational inference. This approach improves performance on heterogeneous datasets while enabling sparser communication, yet it assumes shared task types across clients.

Personalized FL algorithms such as the MTFL method proposed in [37] improve user-level performance by introducing non-federated layers (e.g., BatchNorm or patch layers) to shared DNNs. These methods treat client models as adaptations of a single task rather than distinct tasks and optimize for per-user accuracy using techniques like FedAvg-Adam. Although effective for personalization, such approaches do not model task-level heterogeneity.

In contrast, recent works have started to tackle true MTFL; examples include FedBone [42], a hierarchical FMTL framework that addresses task heterogeneity by decoupling model components between the server and clients using gradient projection and split learning. However, it targets large-scale DNNs and assumes sufficient bandwidth and resources for model splitting, limiting its applicability in lightweight FL scenarios.

Most closely aligned with our healthcare perspective, the work reported in [43] applies a deep learning-based federated MTL approach for fMRI analysis. The proposed method leverages a federated contrastive learning feature extractor and a multi-gate mixture-of-experts classifier to jointly diagnose autism spectrum disorder, ADHD, and schizophrenia. While the approach proposed in ref. [43] effectively uses shared representations across tasks through end-to-end neural networks, our framework complements this direction by incorporating classical ML principles, such as primal optimization with task covariance regularization, which further enhances interpretability and communication efficiency.

Despite these advances, the integration of MTFL with classical ML models such as SVMs is largely underexplored. Existing federated SVM-based methods typically assume single-task learning and do not capture inter-task structure or client heterogeneity. To bridge this gap, we propose the FMT-SVM framework for healthcare applications where multiple tasks overlap. Our method enables each client to learn a task-specific SVM classifier while benefiting from task covariance regularization and CNN-based feature extraction. The proposed approach offers improved accuracy, personalization, and communication efficiency, particularly in healthcare domains with diverse label spaces and data sparsity.

2.3. Problem Formulation

Consider a federated learning (FL) environment consisting of K clients, where each client ($k\in \{1,\dots ,K\}$) holds a private dataset (${\mathcal{D}}_k={\left\{(x_i^{\left(k\right)},y_{i,t}^{\left(k\right)})\right\}}_{i=1}^{n_k}$) for a set of T tasks. Each data sample ($x_i^{\left(k\right)}\in {\mathbb{R}}^{H\times W\times 3}$) represents a medical image, and $y_{i,t}^{\left(k\right)}$ denotes the corresponding task-specific label. The tasks include both binary and multi-class classification, reflecting the heterogeneous nature of real-world healthcare systems.

Unlike prior works [31,33], which primarily focused on single-task learning or simple aggregation strategies, the proposed FMT-SVM framework is explicitly designed to support simultaneous learning of multiple task-specific classifiers across all clients. Each image ($x_i^{\left(k\right)}$) is processed by a unified CNN encoder ($f_{\theta }$), producing a latent representation ($z_k^{\left(i\right)}\in {\mathbb{R}}^d$). Each task ($t\in \{1,\dots ,T\}$) is modeled using a linear SVM head parameterized by $w_{k,t}\in {\mathbb{R}}^d$.

This formulation enables each client to learn two SVM classifiers: (i) a binary classifier for binary prediction tasks and (ii) a multi-class classifier for multi-class prediction tasks. The two classifiers share a common feature representation generated by the CNN encoder ($f_{\theta }$) defined as

$$z_k^{\left(i\right)}=f_{\theta }\left(x_k^{\left(i\right)}\right)\in {\mathbb{R}}^d,$$

(1)

where d denotes the dimensionality of the latent feature space. The shared representation ($z_k^{\left(i\right)}$) is jointly optimized with task-specific classifiers to support task specialization while maintaining federated consistency.

The extracted feature embedding ($z_k^{\left(i\right)}$) is passed to task-specific SVM heads:

• Binary task weights: $w_1^{\left(k\right)}\in {\mathbb{R}}^d$;
• Multi-class task weights: $W_2^{\left(k\right)}\in {\mathbb{R}}^{d\times C}$, where C denotes the number of classes.

2.3.1. Local Multi-Task SVM Objective

Given feature embeddings ($z_k^{\left(i\right)}=f_{\theta }\left(x_k^{\left(i\right)}\right)$), the local optimization problem at client k integrates three components: (i) hinge loss for binary and multi-class tasks, (ii) task covariance regularization, and (iii) global consistency regularization.

$$\begin{array}{cc}\hfill \underset{w_1^{\left(k\right)},W_2^{\left(k\right)}}{\min }& C[{\displaystyle \frac{1}{n_k}}\sum _{i=1}^{n_k}\max \left(0,1-y_{i,1}^{\left(k\right)}{w}_1^{\left(k\right)\top }{z}_k^{\left(i\right)}\right)\hfill \\ \hfill & +{\displaystyle \frac{1}{n_k}}\sum _{i=1}^{n_k}\sum _{j\ne y_{i,2}^{\left(k\right)}}\max \left(0,1+z_k^{\left(i\right)\top }\left(W_2^{\left(k\right)}[:,j]-W_2^{\left(k\right)}[:,y_{i,2}^{\left(k\right)}]\right)\right)]\hfill \\ \hfill & +\lambda \mathrm{Tr}\left({\tilde{W}}_k^{\top }{\mathsf{\Sigma }}_k^{-1}{\tilde{W}}_k\right)\hfill \\ \hfill & +\mu \left({\parallel w_1^{\left(k\right)}-{\overline{w}}_1^{\mathrm{global}}\parallel }^2+{\parallel W_2^{\left(k\right)}-{\overline{W}}_2^{\mathrm{global}}\parallel }_F^2\right)\hfill \end{array}$$

(2)

2.3.2. Unified Representation for Regularization

To facilitate regularization and aggregation, task-specific parameters are stacked into a unified matrix:

$${\tilde{W}}_k=\left[w_1^{\left(k\right)}{W}_2^{\left(k\right)}\right]\in {\mathbb{R}}^{d\times (1+C)}.$$

This unified representation enables multi-task regularization via task covariance modeling and simplifies federated aggregation while preserving task-specific inference.

2.4. Workflow of the Proposed FMT-SVM Framework

Algorithm 1 presents the proposed framework in detail, while Figure 1 illustrates the FMT-SVM workflow. For clarity and reproducibility, we summarize the complete federated training process in a step-by-step manner.

Figure 1. Detailed framework of the proposed FMT-SVM approach. Each client trains a local CNN encoder ($f_{{\theta }_k}$) to produce latent embeddings ($z_k^{\left(i\right)}$) and fits task-specific linear SVM heads (${\tilde{W}}_k$). Clients upload only the SVM heads (optionally perturbed using differential privacy settings) to the server, which aggregates them into a global model (${\tilde{W}}_{\mathrm{global}}$) and broadcasts updated parameters. Task covariance and global consistency regularization are applied locally to promote inter-task sharing and federated coherence, while encoder synchronization is performed infrequently to reduce communication overhead.

Here, $k\in \{1,\dots ,K\}$ denotes the client index, and $r\in \{1,\dots ,R\}$ denotes the communication round.

1.

Initialization of Global Models: The server initializes a global model consisting of a CNN-based encoder and multi-task SVM heads. The encoder maps raw input data to latent embeddings $\mathbf{z}$, while the SVM heads perform binary and multi-class predictions. This initialized model is shared with all clients.

2.

Distribution to Clients: At the beginning of each communication round (r), the server distributes the global SVM heads $({\overline{w}}_1^{(r-1)},{\overline{W}}_2^{(r-1)})$ and task covariance matrix (${\mathsf{\Sigma }}^{(r-1)}$) to all clients.

3.

Local Training on Clients: Each client (k) performs local optimization using its private dataset. For each mini-batch, the following occurs:

• The client computes latent representations:

  $$\mathbf{z}=f_{{\theta }_k^{\left(r\right)}}\left(\mathbf{x}\right)$$
• The embeddings are passed to task-specific SVM heads:

  $$w_1^{(k,r)},W_2^{(k,r)}$$
• The local objective consists of three components:

  –

  Hinge loss for binary and multi-class classification;

  –

  Task covariance regularization, i.e.,

  $$\lambda ·\mathrm{Tr}\left({(w_1^{(k,r)})}^{\top }{\mathsf{\Sigma }}^{-1}{w}_1^{(k,r)}+{(W_2^{(k,r)})}^{\top }{\mathsf{\Sigma }}^{-1}{W}_2^{(k,r)}\right);$$

  –

  Consensus regularization, i.e.,

  $$\mu \left({\parallel w_1^{(k,r)}-{\overline{w}}_1^{(r-1)}\parallel }^2+{\parallel W_2^{(k,r)}-{\overline{W}}_2^{(r-1)}\parallel }^2\right).$$

• The encoder parameters (${\theta }_k^{\left(r\right)}$) and SVM heads are updated via gradient-based primal optimization.

4.

Uploading Local Models: After local training, each client uploads its updated SVM heads $(w_1^{(k,r)},W_2^{(k,r)})$ to the server. The encoder parameters remain local.

5.

Global Aggregation on the Server: The server aggregates client SVM heads, i.e.,

$${\overline{w}}_1^{\left(r\right)}=\sum _{k=1}^K{\displaystyle \frac{n_k}{{\sum }_{j=1}^K{n}_j}}{w}_1^{(k,r)},{\overline{W}}_2^{\left(r\right)}=\sum _{k=1}^K{\displaystyle \frac{n_k}{{\sum }_{j=1}^K{n}_j}}{W}_2^{(k,r)}.$$

The server also updates as follows:

$${\mathsf{\Sigma }}^{\left(r\right)}=\mathrm{Cov}\left([W_1^{(1,r)},\dots ,W_K^{(K,r)}]\right).$$

6.

Distribution of Updated Global Heads: The server broadcasts $({\overline{w}}_1^{\left(r\right)},{\overline{W}}_2^{\left(r\right)},{\mathsf{\Sigma }}^{\left(r\right)})$ back to clients.

7.

Evaluation of the Global Model: The server evaluates

$${\mathbf{z}}_{\mathrm{test}}=f_{\theta }\left({\mathbf{x}}_{\mathrm{test}}\right),$$

followed by prediction using global heads.

Algorithm 1 Federated Multi-Task SVM with Personalized CNN Encoders

Require: Number of clients N, communication rounds R, local epochs E, learning rate $\eta$, SVM penalty C Require: Regularization parameters $\lambda$ (task coupling), $\mu$ (global consistency) Require: Initial encoder parameters ${\left\{{\theta }_i^0\right\}}_{i=1}^N$, global SVM heads $W_{\mathrm{global}}^0$, task covariance ${\mathsf{\Sigma }}^0$ Require: Client datasets ${\left\{{\mathcal{D}}_i\right\}}_{i=1}^N$, ${\mathcal{D}}_i=\left\{(x_j,y_j^{\left(1\right)},y_j^{\left(2\right)})\right\}$ Ensure: Personalized encoders ${\left\{{\theta }_i^R\right\}}_{i=1}^N$, global heads $W_{\mathrm{global}}^R$, task covariance ${\mathsf{\Sigma }}^R$    1: for $r=1$ to R do    2:       Server broadcasts $W_{\mathrm{global}}^{(r-1)}$ and ${\mathsf{\Sigma }}^{(r-1)}$    3:       for each client $i=1,\dots ,K$ in parallel do    4:          Initialize $W_i^{\left(r\right)}\leftarrow W_{\mathrm{global}}^{(r-1)}$    5:          for $e=1$ to E do    6:              for each minibatch $(x_j,y_j^{\left(1\right)},y_j^{\left(2\right)})\in {\mathcal{D}}_i$ do    7:                    $z_j\leftarrow f_{{\theta }_i^{(r-1)}}\left(x_j\right)$    8:                    ${\mathcal{L}}_j^{\left(1\right)}\leftarrow \max (0,1-y_j^{\left(1\right)}{w}_i^{\left(1\right)\top }{z}_j)$    9:                    ${\mathcal{L}}_j^{\left(2\right)}\leftarrow {\sum }_{k\ne y_j^{\left(2\right)}}\max (0,1+w_{ik}^{\left(2\right)\top }{z}_j-w_{i{y}_j^{\left(2\right)}}^{\left(2\right)\top }{z}_j)$    10:                  ${\mathcal{L}}_{\mathrm{task}}\leftarrow {\sum }_j\left({\mathcal{L}}_j^{\left(1\right)}+{\mathcal{L}}_j^{\left(2\right)}\right)$    11:                  ${\mathcal{R}}_{\mathrm{cov}}\leftarrow \lambda \mathrm{Tr}\left(W_i^{\top }{\mathsf{\Sigma }}^{-1}{W}_i\right)$    12:                  ${\mathcal{R}}_{\mathrm{cons}}\leftarrow \mu {\parallel W_i-W_{\mathrm{global}}^{(r-1)}\parallel }_F^2$    13:                  ${\mathcal{L}}_i\leftarrow {\mathcal{L}}_{\mathrm{task}}+{\mathcal{R}}_{\mathrm{cov}}+{\mathcal{R}}_{\mathrm{cons}}$    14:                  ${\theta }_i^{\left(r\right)}\leftarrow {\theta }_i^{(r-1)}-\eta {\nabla }_{{\theta }_i}{\mathcal{L}}_i$    15:                  $W_i^{\left(r\right)}\leftarrow W_i^{\left(r\right)}-\eta {\nabla }_{W_i}{\mathcal{L}}_i$    16:            end for    17:        end for    18:        Client i uploads $W_i^{\left(r\right)}$    19:     end for    20:     $W_{\mathrm{global}}^{\left(r\right)}\leftarrow {\sum }_{i=1}^N{\lambda }_i{W}_i^{\left(r\right)}$    21:     Compute mean model: ${\overline{W}}^{\left(r\right)}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{W}_i^{\left(r\right)}$    22:     for $i=1$ to N do    23:           $Z_i^{\left(r\right)}\leftarrow W_i^{\left(r\right)}-{\overline{W}}^{\left(r\right)}$    24:     end for    25:     ${\mathsf{\Sigma }}^{\left(r\right)}\leftarrow {\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\left(Z_i^{\left(r\right)}\right)}^{\top }{Z}_i^{\left(r\right)}$    26:     ${\mathsf{\Sigma }}^{\left(r\right)}\leftarrow {\mathsf{\Sigma }}^{\left(r\right)}+ϵI$    27: end for

2.5. CNN-Based Feature Extraction

In our FMT-SVM framework, each client employs a lightweight unified CNN encoder, denoted by $f_{\theta }$, to extract compact and informative representations from local input data. This encoder maps raw input images into a latent feature space that serves as the basis for task-specific classifiers, supporting both binary and multi-class classification tasks. While the encoder architecture is similar across clients, each client maintains its own encoder parameters ($F_{{\theta }_k}$) to extract meaningful features adaptive to its data distribution, supporting improved performance in heterogeneous federated learning settings.

Let each input image can be denoted by $x^{\left(i\right)}\in {\mathbb{R}}^{H\times W\times C}$, where H, W, and C represent the height, width, and number of channels of the image, respectively. The unified local encoder maps the input image into a latent embedding space as follows:

$$z_k^{\left(i\right)}=f_{{\theta }_k}\left(x^{\left(i\right)}\right)\in {\mathbb{R}}^d$$

(3)

where:

• $f_{{\theta }_k}$ is the CNN encoder parameterized by $\theta$ for client k;
• $z_t^{\left(k\right)}$ is the learned feature embedding for the i-th image from client k;
• d is the dimensionality of the latent space.

The encoder consists of two convolutional layers followed by max pooling, a flattening operation, and a fully connected layer, which are discussed below.

2.5.1. Convolution and Activation Layers

The encoder begins with two 2D convolutional layers, which serve to extract local spatial features from the input image. Each convolution uses a kernel of size $3\times 3$ with padding of 1 to preserve spatial dimensions before pooling. The first layer maps from 3 input channels (RGB) to 16 filters, and the second layer maps from 16 to 32 filters. Formally,

$$\begin{array}{cc}\hfill h^{\left(1\right)}& =\sigma \left(\mathrm{Conv}2\mathrm{D}(I_t^{\left(i\right)};W_1,b_1)\right),\hfill \\ \hfill h^{\left(2\right)}& =\sigma \left(\mathrm{Conv}2\mathrm{D}(p^{\left(1\right)};W_2,b_2)\right).\hfill \end{array}$$

(4)

where $W_1,W_2$ are the convolutional kernels, $b_1,b_2$ are the respective biases, and $\sigma (·)$ denotes an element-wise ReLU activation function. The ReLU function is defined as follows:

$$\sigma \left(x\right)=max(0,x)$$

(5)

We specifically chose ReLU as the activation function due to its computational simplicity and empirical effectiveness. This activation introduces non-linearity into the model. Zeroing out all negative values and preserving positive ones helps to induce sparsity in activations; reduces the likelihood of vanishing gradients during backpropagation; and enables the encoder to learn complex, hierarchical feature representations from the input images.

2.5.2. Max-Pooling Layers

Each convolutional layer is followed by a $2\times 2$ max-pooling layer with a stride value of 2, which reduces the spatial resolution by half. This layer improves the translational invariance and also reduces the computational overhead. The pooled feature maps are given by

$$\begin{array}{c}{p}^{\left(1\right)}={\mathrm{MaxPool}}_{2\times 2,s=2}\left(h^{\left(1\right)}\right),\hfill \\ p^{\left(2\right)}={\mathrm{MaxPool}}_{2\times 2,s=2}\left(h^{\left(2\right)}\right)\hfill \end{array}$$

(6)

2.5.3. Flattening and Fully Connected Projection

The output of the second max-pooling layer ($p^{\left(2\right)}\in {\mathbb{R}}^{32\times 16\times 16}$) is then flattened into a one-dimensional vector, which is then passed through a fully connected layer:

$$z_k^{\left(i\right)}=W_f·\mathrm{Flatten}\left(p^{\left(2\right)}\right)+b_f$$

(7)

where $W_f\in {\mathbb{R}}^{d\times 8192}$ and $b_f\in {\mathbb{R}}^d$. The output ($z_k^{\left(i\right)}\in {\mathbb{R}}^d$) represents a compact latent embedding that captures the high-level spatial features of the input image.

Importantly, each client employs a unified local encoder that extracts a common latent representation for multiple related tasks (e.g., binary classification and multi-class classification). This encoder enables the learning of a unified feature space per client, which is then fed into separate task-specific SVM heads. While the encoder is shared between tasks locally, each client maintains its own personalized encoder parameters, allowing for adaptation to local data distributions.

2.6. Task-Specific SVM Heads

The latent embedding ($z_k^{\left(i\right)}\in {\mathbb{R}}^d$) extracted by the CNN encoder, is then fed to a set of task-specific linear classifiers configured for either binary or multi-class prediction tasks. In our framework, these heads are implemented as linear SVMs, one for each classification task $t\in \{1,\dots ,T\}$, where each SVM head is parameterized in its primal form by a weight vector ($w_{k,t}\in {\mathbb{R}}^d$) and a bias term ($b_{k,t}\in \mathbb{R}$). Importantly, these parameters represent the hyperplane coefficients learned by the SVM, not the raw support vectors or training samples. This maintains privacy and also avoids the storage and transfer of potentially large sets of support vectors, significantly reducing communication cost in the federated setting.

The complete set of task-specific classifiers for client k is represented as follows:

$$W_k=[w_{k,1},w_{k,2},\dots ,w_{k,T}]\in {\mathbb{R}}^{d\times T}$$

(8)

The decision score for the t-th task on the i-th input is computed as follows:

$${\widehat{y}}_{k,t}^{\left(i\right)}=w_{k,t}^{\top }{z}_k^{\left(i\right)}+b_{k,t}$$

(9)

These decision scores are used within the hinge loss formulation defined in the local multi-task SVM objective.

By using linear SVMs for classification, we leverage a margin-based objective that is well-suited for structured tasks and offers strong generalization, especially in low-data or imbalanced regimes.

2.7. Task Covariance Regularization

We employ task covariance regularization, which extends classical multi-task learning by regularizing task-specific predictors using the trace term ($\mathrm{tr}\left(W\mathsf{\Omega }{W}^{\top }\right)$, where $\mathsf{\Omega }$ encodes task relationships). In our FMT-SVM framework, this maps to the regularization term:

$${\mathcal{R}}_{\mathrm{task}}\left(W\right)=\mathrm{tr}\left(W{\mathsf{\Sigma }}^{-1}{W}^{\top }\right)$$

(10)

where $W\in {\mathbb{R}}^{d\times T}$ denotes the matrix of task-specific model parameters and $\mathsf{\Sigma }\in {\mathbb{R}}^{T\times T}$ represents the task covariance matrix capturing inter-task relationships. The inverse covariance (${\mathsf{\Sigma }}^{-1}$) acts as a precision matrix, enforcing stronger coupling between highly correlated tasks.

To explicitly model task dependencies in the federated setting, $\mathsf{\Sigma }$ is estimated from client updates in each communication round. Let $W_i^{\left(r\right)}$ denote the task-specific parameter matrix at client i during round r, and let N be the total number of clients. We first compute the global mean model:

$${\overline{W}}^{\left(r\right)}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{W}_i^{\left(r\right)}$$

(11)

We then define the centered client updates (Z) as follow:

$$Z_i^{\left(r\right)}=W_i^{\left(r\right)}-{\overline{W}}^{\left(r\right)}$$

(12)

The task covariance matrix is then computed as the empirical covariance:

$${\mathsf{\Sigma }}^{\left(r\right)}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(Z_i^{\left(r\right)}\right)}^{\top }{Z}_i^{\left(r\right)}$$

(13)

This is equivalent to $\mathrm{Cov}(W_1^{\left(r\right)},\dots ,W_N^{\left(r\right)})$ used in Algorithm 1.

To ensure numerical stability and invertibility, we regularize the covariance matrix as follows:

$${\mathsf{\Sigma }}^{\left(r\right)}\leftarrow {\mathsf{\Sigma }}^{\left(r\right)}+ϵI$$

(14)

where $ϵ>0$ is a small constant and I is the identity matrix.

This formulation enables dynamic adaptation of task relationships across communication rounds, allowing the model to capture evolving inter-task dependencies under heterogeneous data distributions. The regularization encourages related tasks to share statistical strength while preserving task-specific decision boundaries. When $\mathsf{\Sigma }$ reduces to the identity matrix, the formulation becomes equivalent to standard ${\mathcal{l}}_2$ regularization. In contrast, a low-rank $\mathsf{\Sigma }$ induces a shared latent subspace across tasks, promoting stronger coupling among correlated tasks.

2.8. Global Consistency Regularization

In our proposed federated multi-task settings, there must be an alignment between the local models (${\tilde{W}}_k$) learned by individual clients (k) and the global model (${\tilde{W}}_{\mathrm{global}}$), which is crucial in preventing model drifting and promoting generalization. To achieve this, we introduce a global consistency regularization term in the local objective of each client, which penalizes deviation between the local task-specific classifier (${\tilde{W}}_k$) and the global model (${\tilde{W}}_{\mathrm{global}}$).

Formally, the global consistency penalty is defined using the squared Frobenius norm as follow:

$${\mathcal{R}}_{\mathrm{global}}=\mu ·{\parallel {\tilde{W}}_k-{\tilde{W}}_{\mathrm{global}}\parallel }_F^2$$

Here:

• ${\tilde{W}}_k\in {\mathbb{R}}^{d\times (1+C)}$ is the unified classifier matrix at client k, obtained by concatenating both binary and multi-class heads;
• ${\tilde{W}}_{\mathrm{global}}\in {\mathbb{R}}^{d\times (1+C)}$ is the globally aggregated model shared by the server;
• $\mu >0$ is a regularization hyperparameter that controls the strength of alignment across clients.

This term ensures that clients do not overfit to their private datasets and also facilitates stable convergence during training. From an optimization perspective, this term also contributes strong convexity to the objective function, improving theoretical convergence guarantees.

2.9. Global Aggregation

After the local training phase, the server updates the global model by performing a weighted averaging of the client models. Let ${\displaystyle \frac{n_k}{{\sum }_{j=1}^K{n}_j}}$ be the relative weight of client k, where $n_k$ is the number of samples at client k. For any parameter ($W\in \{w_1,b_1,W_2,b_2\}$), the global update is

$${\tilde{W}}_{\mathrm{global}}=\sum _{k=1}^K{\displaystyle \frac{n_k}{{\sum }_{j=1}^K{n}_j}}{\tilde{W}}_k.$$

(15)

No dual variables are exchanged, and only primal task weights (${\overline{W}}_k$) are communicated. This reduces overhead and aligns with common federated optimization protocols such as FedAvg and FedProx. Each client independently updates its local model by minimizing Equation (2) using stochastic gradient descent. Gradients are backpropagated through both the SVM loss and the CNN encoder. In each communication round, the global server broadcasts the current $W_{\mathrm{global}}$ to all clients. The process is repeated until convergence.

2.10. Primal–Dual Formulation for FMT-SVM

To provide a theoretically grounded perspective for optimization in federated multi-task learning, we adopt a primal–dual viewpoint of the FMT-SVM formulation. This not only clarifies the structure of the federated objective but also highlights the distributed coupling across clients and heterogeneous tasks.

2.10.1. Primal Objective

As introduced earlier, each client ($k\in \{1,\dots ,K\}$) holds a private dataset (${\mathcal{D}}_k={\left\{(x_k^{\left(i\right)},y_k^{\left(i\right)})\right\}}_{i=1}^{n_k}$, where $x_k^{\left(i\right)}$ is a raw input image and $y_k^{\left(i\right)}\in {\mathbb{R}}^T$ encodes task-specific labels for T prediction tasks). The image is transformed into a latent representation ($z_k^{\left(i\right)}=f_{\theta }\left(x_k^{\left(i\right)}\right)\in {\mathbb{R}}^d$) via the encoder ($f_{\theta }$).

Each client then learns a unified classifier (${\tilde{W}}_k=\left[w_1^{\left(k\right)}{W}_2^{\left(k\right)}\right]\in {\mathbb{R}}^{d\times (1+C)}$, where $w_1^{\left(k\right)}$ corresponds to the binary task head and $W_2^{\left(k\right)}$ corresponds to a multi-class head with C classes). The local primal optimization objective is expressed as follows:

$$\begin{array}{cc}\hfill \underset{{\tilde{W}}_k}{\min }& {\displaystyle \frac{1}{n_k}}\sum _{i=1}^{n_k}\underset{\mathrm{Binary}\mathrm{Hinge}\mathrm{Loss}}{\underbrace{\max \left(0,1-y_{i,1}^{\left(k\right)}·w_1^{\left(k\right)\top }{z}_k^{\left(i\right)}\right)}}\hfill \\ \hfill & +{\displaystyle \frac{1}{n_k}}\sum _{i=1}^{n_k}\sum _{j\ne y_{i,2}^{\left(k\right)}}\underset{\mathrm{Multi-class}\mathrm{Hinge}\mathrm{Loss}}{\underbrace{\max \left(0,1+z_k^{\left(i\right)\top }(W_2^{\left(k\right)}[:,j]-W_2^{\left(k\right)}[:,y_{i,2}^{\left(k\right)}])\right)}}\hfill \\ \hfill & +\lambda ·\mathrm{Tr}\left({\tilde{W}}_k^{\top }{\mathsf{\Sigma }}_k^{-1}{\tilde{W}}_k\right)+\mu ·{\parallel {\tilde{W}}_k-{\tilde{W}}_{\mathrm{global}}\parallel }_F^2\hfill \end{array}$$

where ${\mathsf{\Sigma }}_k\in {\mathbb{R}}^{(1+C)\times (1+C)}$ is a task covariance matrix modeling inter-task relationships on client k and ${\tilde{W}}_{\mathrm{global}}$ is the global reference model broadcast from the server.

2.10.2. Dual View of the Local Hinge Loss

For each binary or multi-class task, the hinge loss can be interpreted in its dual form. For the binary SVM head, the dual optimization introduces Lagrange multipliers (${\alpha }_k^{\left(i\right)}\in [0,C]$), leading to the dual problem:

$$\begin{array}{c}\hfill \underset{{\alpha }_k}{\max }\sum _{i=1}^{n_k}{\alpha }_k^{\left(i\right)}-{\displaystyle \frac{1}{2}}\sum _{i,j=1}^{n_k}{\alpha }_k^{\left(i\right)}{\alpha }_k^{\left(j\right)}{y}_{i,1}^{\left(k\right)}{y}_{j,1}^{\left(k\right)}⟨z_k^{\left(i\right)},z_k^{\left(j\right)}⟩,\\ \hfill \mathrm{s}.\mathrm{t}.0\le {\alpha }_k^{\left(i\right)}\le C\end{array}$$

(16)

This dual view shows that only support vectors contribute to the classifier, offering potential benefits in sparsification or communication-efficient updates. While our FMT-SVM method operates only in the primal form for computational reasons, the dual interpretation remains valuable for theoretical analysis.

2.10.3. Global Coupling and Regularization via Consensus

The final two terms in Equation (16) introduce global coupling across clients:

• The trace regularization term encourages inter-task parameter sharing based on the learned task similarity matrix (${\mathsf{\Sigma }}_k$).
• The global consistency term ($\parallel {\tilde{W}}_k-{\tilde{W}}_{\mathrm{global}}{\parallel }_F^2$) ensures local models align with the global objective across federated rounds.

In a dual interpretation, these regularizers imply implicit constraints on the dual variables, encouraging soft consensus across tasks and clients.

2.10.4. Why We Adopt a Primal-Only Strategy

Despite the interpretability of the dual form, FMT-SVM is optimized in the primal space for the following reasons:

• CNN-based embeddings make the kernel matrix $⟨z_k^{\left(i\right)},z_k^{\left(j\right)}⟩$ non-static and computationally expensive to maintain.
• End-to-end training with deep networks is more naturally expressed and implemented in the primal form.
• Communication efficiency is better preserved without transmitting high-dimensional dual variables or kernel matrices.

Thus, we adopt stochastic primal optimization using SGD while retaining theoretical soundness through the convexity of the SVM-based loss.

2.11. Baseline Models

To evaluate the effectiveness of the proposed FMT-SVM framework, we compare it against two representative baseline models:

FedPureCNN: An end-to-end federated CNN where the encoder and classifier are trained jointly and the complete set of parameters is exchanged between clients and the server at each communication round. This baseline represents the conventional high-performance FL approach, but it suffers from high communication overhead because it transmits millions of parameters per round.

FedPureSVM: A federated linear SVM model trained directly on raw input features without any shared deep feature extractor. Only the linear SVM weights are exchanged between clients and the server, making this approach communication-efficient but lacking the representation power of deep neural features.

Both baselines are implemented using the standard FedAvg aggregation protocol to ensure a fair and consistent optimization setting across all methods. These baselines allow us to contrast FMT-SVM’s performance against both a computationally and communication-intensive model (FedPureCNN) and a lightweight, shallow model (FedPureSVM), thereby illustrating the trade-off between accuracy and communication cost.

2.12. Computational Complexity and Scalability Analysis

The computational and communication costs of the compared methods differ significantly due to their model structures and parameter-sharing strategies. In addition, scalability with respect to the number of clients is a critical factor in federated environments. Table 2 presents a comparison of the computational and communication complexity of the considered methods, highlighting the scalability advantages of the proposed FMT-SVM framework. Here, E denotes the number of local training epochs, K is the total number of participating clients, $N_k$ is the number of samples at client k, R is the number of communication rounds, d represents the feature dimension, P denotes the total number of parameters in the CNN model, and $C_{cnn}$ represents the computational cost of a forward and backward pass of the CNN per sample. As shown in Table 2, FMT-SVM achieves a significant reduction in communication complexity compared to CNN-based methods while maintaining high model capacity. FedPureCNN requires full CNN training at each client, resulting in a computational complexity of $\mathcal{O}(E·N_k·C_{\mathrm{cnn}})$. It also incurs a high communication cost of $\mathcal{O}(R·K·P)$, as all model parameters are exchanged at every round. As the number of clients (K) increases, the communication overhead grows linearly, making it less scalable for large federated systems.

Table 2. Comparison of computational and communication complexity of different methods.

Method	Local Computation	Communication Cost	Model Capacity
FedPureCNN	$\mathcal{O}\left(E{N}_k{C}_{\mathrm{cnn}}\right)$	$\mathcal{O}\left(RKP\right)$	High
FedPureSVM	$\mathcal{O}\left(E{N}_{k}d\right)$	$\mathcal{O}\left(RKd\right)$	Low
FMT-SVM (Proposed)	$\mathcal{O}\left(E{N}_k{C}_{\mathrm{cnn}}\right)$	$\mathcal{O}\left(RKTd\right)$	High

FedPureSVM is computationally lightweight, with a complexity of $\mathcal{O}(E·N_k·d)$ and communication cost of $\mathcal{O}(R·K·d)$, since only linear SVM weights are transmitted. While it scales well in terms of communication, its limited representational capacity restricts performance on complex tasks.

In contrast, the proposed FMT-SVM framework combines the strengths of both approaches. At each client, the dominant cost arises from CNN-based feature extraction, with a complexity of $\mathcal{O}(E·N_k·C_{\mathrm{cnn}})$, while the multi-task SVM head adds a smaller overhead of $\mathcal{O}(E·N_k·d)$. Task covariance regularization further introduces a minor cost of $\mathcal{O}\left(T^{2}d\right)$.

On the server side, aggregation of SVM heads across K clients requires $\mathcal{O}(K·T·d)$, and updating the task covariance matrix incurs $\mathcal{O}(K·T^{2}d)$, both of which remain efficient, even as the number of clients increases.

Importantly, FMT-SVM significantly reduces communication overhead by transmitting only the SVM head parameters, resulting in a communication complexity of $\mathcal{O}(R·K·T·d)$, compared to $\mathcal{O}(R·K·P)$ for CNN-based methods (where $P\gg d$). This reduction makes FMT-SVM more scalable in large federated settings.

Furthermore, due to the convex optimization of the SVM head and the structured task coupling, FMT-SVM is expected to exhibit stable convergence behavior, even under heterogeneous (non-IID) client distributions, which is a known challenge in federated learning.

Overall, FMT-SVM achieves an effective trade-off by maintaining strong representation learning while substantially reducing communication cost, making it suitable for large-scale and resource-constrained federated environments.

2.13. Convergence Discussion

We provide a brief theoretical analysis of the convergence behavior of the proposed FMT-SVM framework.

Theorem 1 (Convergence of FMT-SVM).

Assume that the local loss functions are convex and have bounded gradients and that the task covariance matrix satisfies ${\mathsf{\Sigma }}^{-1}\succ 0$ with $\mu >0$. Then, the local objective at each client is strongly convex and admits a unique minimizer. Under standard assumptions on learning-rate selection, the iterative updates at each client converge to this minimizer. Furthermore, with periodic aggregation across communication rounds, the global model converges to a stationary point of the overall objective, even in the presence of non-IID data distributions.

Proof. 

The hinge loss is convex, and the inclusion of the covariance regularization term ($\mathrm{tr}\left(W{\mathsf{\Sigma }}^{-1}{W}^{\top }\right)$), along with the proximal term ($\parallel W_i-W_{\mathrm{global}}{\parallel }_F^2$), ensures strong convexity of the objective. This guarantees the uniqueness of the local solution. The aggregation step follows standard federated optimization schemes, and convergence to a stationary point can be established using arguments similar to those in [8,48]. The proximal term further bounds client drift, ensuring stability under heterogeneous data. □

2.14. Communication Cost Analysis

A key advantage of the proposed FMT-SVM framework is its communication-efficient parameter-sharing strategy. Unlike conventional federated learning approaches such as FedAvg or FedProx, which require the transmission of the entire neural network model between the server and clients, FMT-SVM transmits only the task-specific SVM heads while keeping the CNN encoder local to each client. For each client k with $T=2$ tasks, the transmitted parameters consist of the following:

• Binary classification head:

  –

  Weight vector ($w_1\in {\mathbb{R}}^d$);

  –

  Bias scalar ($b_1\in \mathbb{R}$).

• Multi-class classification head:

  –

  Weight matrix ($W_2\in {\mathbb{R}}^{d\times C}$);

  –

  Bias vector ($b_2\in {\mathbb{R}}^C$).

Thus, the communication cost per client per round is expressed as follows:

$$\mathrm{CommCost}=\left[d+1+d·C+C\right]\times s_{\mathrm{float}},$$

(17)

where $s_{\mathrm{float}}=4$ bytes for a 32-bit floating-point representation.

The FedPureCNN transmits the entire CNN model (including the encoder and classifier) at each communication round. The communication cost is $\mathcal{O}\left(P\right)$, where P is the total number of network parameters. In our implementation, this corresponds to approximately 4.02 MB per round per client.

Similarly, FedPureSVM transmits only a single-task linear SVM model, resulting in a communication cost of $\mathcal{O}\left(d\right)$ per round. While this is highly communication-efficient, it lacks the representational power of deep feature extractors.

In contrast, FMT-SVM transmits only the multi-task SVM heads, resulting in a communication cost of $\mathcal{O}(T·d)$. In our implementation, this corresponds to approximately 2.06 KB per round per client.

Compared to FedPureCNN, FMT-SVM achieves an approximately 400× reduction in communication overhead while still maintaining high model capacity due to the use of CNN-based feature extraction. Compared to FedPureSVM, FMT-SVM provides significantly improved representational power with only a marginal increase in communication cost.

Furthermore, since FMT-SVM transmits only model parameters (i.e., hyperplane coefficients $(w_{k,t},b_{k,t})$) and not raw data or support vectors, the communication cost is independent of dataset size and inherently privacy-preserving.

This trade-off between communication efficiency and model expressiveness makes FMT-SVM particularly suitable for large-scale and bandwidth-constrained federated learning environments.

3. Experiments and Results

In this section, we empirically evaluate the performance of the proposed FMT-SVM framework on real-world healthcare prediction tasks. Furthermore, we compare FMT-SVM with several baseline methods to assess its effectiveness in terms of predictive performance, communication efficiency, and privacy preservation.

3.1. Experimental Setup

3.1.1. Datasets

We conduct extensive experiments across multiple federated configurations to evaluate the performance of the proposed FMT-SVM framework on two real-world multi-task medical imaging datasets from the MedIMeta dataset [49], a multi-task medical image dataset that represents different modalities and disease prediction scenarios. We chose these datasets to show the efficacy and generalizability of FMT-SVM over different clinical tasks, viz., pediatric pneumonia detection from chest X-rays [49] and breast cancer diagnosis from ultrasound images [49].

1.

Pediatric Pneumonia Dataset: We utilize the publicly available Pediatric Pneumonia Dataset from MedIMeta [49], which comprises 5856 chest X-ray images, as illustrated in Figure 2a. The dataset has two tasks: (1) Pneumonia presence: A binary classification task with two classes, viz., NORMAL and PNEUMONIA. (2) Disease class: A multi-class classification task with three classes, viz., NORMAL, BACTERIA, and VIRUS. This dataset tests FMT-SVM’s ability to handle large-scale image-based healthcare data and class imbalance across multiple pulmonary diseases.

Figure 2. A glimpse of the datasets used in this study. (a) Pediatric Pneumonia Dataset; (b) Breast Ultrasound Dataset [49].

2.

Breast Ultrasound Dataset: We utilize the publicly available Breast Ultrasound Dataset from MedIMeta [49], which comprises 780 breast ultrasound images, as illustrated in Figure 2b. The dataset has two tasks: (1) Malignancy Detection: A binary classification task with two classes, viz., no malignant finding and malignant finding. (2) Disease class: A multi-class classification task having three classes, viz. normal, benign, and malignant.

This dataset tests FMT-SVM’s ability to generalize to smaller, noisier, and heterogeneous datasets common in resource-constrained healthcare settings.

3.1.2. Federated Environment Simulation

To simulate a federated learning setting for healthcare, we conducted systematic experiments:

• Data is partitioned across $K=5$ simulated clients, each containing unique, non-IID samples, using the Dirichlet distribution, which introduces varying levels of statistical heterogeneity among clients. Specifically, the label distribution for each client is sampled from a Dirichlet distribution with a concentration parameter of $\alpha \in 0.3,0.5,0.9$, where smaller values of $\alpha$ result in more skewed (non-IID) distributions and larger values approach uniformity (IID). Figure 3 and Figure 4 illustrate the client-wise label distributions for both binary and multi-class tasks under different $\alpha$ values. When $\alpha =0.3$, the data distribution is highly skewed, with certain clients dominated by specific classes, representing a strongly non-IID setting. As $\alpha$ increases to $0.5$ and $0.9$, the class distributions become progressively more balanced across clients, approaching an IID scenario. This controlled variation enables a systematic evaluation of the proposed model under different levels of statistical heterogeneity.
• Local epochs (E $\in \{1,5,10\}$) are used to explore trade-offs in local computation versus communication.
• Global communication rounds (R $\in \{10,20\}$) are used to analyze the effect of communication budget on convergence.

Figure 3. Pediatric Pneumonia Dataset: Client data distribution with number of samples belonging to both tasks for each client with varying $\alpha$ values.

Figure 4. Breast Ultrasound Dataset: Client data distribution with number of samples belonging to both tasks for each client with varying $\alpha$ values.

3.1.3. Tasks

We evaluate FMT-SVM on $T=2$ disease classification tasks: (1) binary classification task and (2) multi-class classification task, on two datasets with different medical image modalities.

3.2. Hyperparameter Tuning

In this section, we describe the selection process for the key hyperparameters in the proposed FMT-SVM framework. These hyperparameters govern the strength of regularization components and margin constraints and ultimately influence the model’s generalization and convergence.

Key Hyperparameters

We consider the following core hyperparameters:

• $\lambda$: Weight of the task covariance regularization, controlling how strongly tasks are encouraged to share information. This term penalizes deviation from the inverse task similarity matrix and helps encode inter-task relationships.

  $$\lambda ·\mathrm{Tr}\left(W^{\top }{\mathsf{\Sigma }}^{-1}W\right)$$
• $\mu$: Weight of the global consistency regularization, enforcing alignment between local client models ($W_k$) and the global model ($W_{\mathrm{global}}$).

  $$\mu ·\parallel W_k-W_{\mathrm{global}}{\parallel }^2$$
• C: SVM margin penalty parameter used in both binary and multi-class hinge loss formulations. Higher values penalize misclassifications more strictly, potentially at the cost of generalization.

We performed a grid search over a range of values for each hyperparameter while keeping the others fixed. Specifically, we set

$$\begin{array}{c}\hfill \lambda \in \{0.0,0.01,0.1,1.0\},\mu \in \{0.01,0.1,1.0\},\\ \hfill C\in \{0.01,0.1,1.0\}.\end{array}$$

(18)

Each configuration was evaluated on both datasets with $\alpha =0.5$ using held-out validation data across all clients. Performance was evaluated for E = 1 and R = 10 and assessed in terms of accuracy for both binary and multi-class tasks, as illustrated in Table 3.

Table 3. Grid-search results showing the effect of varying the $\lambda$, $\mu$, and C hyperparameters on Task 1 (Binary) and Task 2 (Multi-class) classification performance in terms of accuracy.

Hyperparameters			Pediatric Pneumonia Dataset		Breast Ultrasound Dataset
$\mathit{\lambda }$	$\mathit{\mu }$	$\mathit{C}$	Task 1	Task 2	Task 1	Task 2
0.0	0.0	0.01	0.8752	0.6530	0.6667	0.5000
0.0	0.0	0.1	0.9368	0.7487	0.6667	0.5000
0.0	0.0	1.0	0.9538	0.8137	0.7692	0.6154
0.0	0.01	0.01	0.9368	0.7658	0.6667	0.5000
0.0	0.01	0.1	0.9487	0.8034	0.8205	0.6538
0.0	0.01	1.0	0.8222	0.7026	0.8077	0.7051
0.0	0.1	0.01	0.9607	0.7060	0.7308	0.3718
0.01	0.0	0.01	0.9487	0.7487	0.6667	0.6538
0.01	0.0	0.1	0.9145	0.8137	0.8462	0.6026
0.01	0.0	1.0	0.9692	0.8188	0.7949	0.6282
0.01	0.01	0.01	0.9470	0.6821	0.8077	0.6923
0.01	0.01	0.1	0.8274	0.6632	0.8590	0.6164
0.01	0.01	1.0	0.8667	0.6308	0.8077	0.6538
0.01	0.1	0.01	0.9504	0.6803	0.8462	0.5000
0.01	0.1	0.1	0.9590	0.8068	0.8718	0.6026
0.01	0.1	1.0	0.9658	0.8359	0.9103	0.7179
0.1	0.0	0.01	0.9692	0.7231	0.8846	0.6667
0.1	0.0	0.1	0.9573	0.7590	0.8718	0.6538
0.1	0.0	1.0	0.9658	0.8034	0.8590	0.7179
0.1	0.01	0.01	0.9521	0.6581	0.7051	0.4744
0.1	0.01	0.1	0.9573	0.8000	0.8846	0.6923
0.1	0.01	1.0	0.9709	0.8342	0.9231	0.7692
0.1	0.1	0.01	0.9368	0.7077	0.8077	0.6154
0.1	0.1	0.1	0.9043	0.6906	0.8974	0.7179
0.1	0.1	1.0	0.9761	0.8103	0.9359	0.7821
1.0	0.0	0.01	0.7538	0.4769	0.6538	0.5897
1.0	0.0	0.1	0.9299	0.6667	0.8718	0.7179
1.0	0.0	1.0	0.9556	0.7897	0.9231	0.7564
1.0	0.01	0.01	0.8171	0.6393	0.7308	0.5000
1.0	0.01	0.1	0.9316	0.6684	0.9103	0.6282
1.0	0.01	1.0	0.9282	0.7726	0.9359	0.7692
1.0	0.1	0.01	0.8427	0.6051	0.8333	0.7051
1.0	0.1	0.1	0.9350	0.6752	0.9103	0.7179
1.0	0.1	1.0	0.9333	0.7675	0.9359	0.7436

Note: Bold values indicate the best performance (highest accuracy) across hyperparameter combinations.

The optimal configuration selected for final experiments is summarized in Table 4.

Table 4. Best hyperparameter configuration used in final experiments for both datasets.

Dataset	Hyperparameter
	$\mathit{\lambda }$	$\mathit{\mu }$	$\mathit{C}$
Pediatric Pneumonia Dataset	0.1	0.01	1.0
Breast Ultrasound Dataset	0.1	0.1	1.0

3.3. Results

Table 5 presents a comprehensive evaluation of our proposed FMT-SVM framework with CNN encoders. We assess the impact of data heterogeneity (Dirichlet $\alpha$), local training epochs (E), and global communication rounds (R) jointly on both binary (Task 1) and multi-class (Task 2) classification tasks. Each configuration was evaluated across five clients with a fixed CNN encoder and separate multi-task heads. The model was trained on both the pneumonia dataset (binary: pneumonia detection; multi-class: disease type) and the breast ultrasound dataset (binary: malignancy detection; multi-class: disease class). The results offer insight into convergence behavior, robustness to non-IID data, and stability across varying federated configurations.

Table 5. Comprehensive performance of the proposed FMT-SVM framework across different data heterogeneity levels ($\alpha$), local epochs (E), and global rounds (R) for both the Pediatric Pneumonia and Breast Ultrasound datasets.

Dataset	$\mathit{\alpha }$	E	R	Task1Acc	Task1AUC	Task2Acc	Task2AUC
Pediatric Pneumonia Dataset	0.9	1	10	0.9607	0.9895	0.8085	0.9201
		1	20	0.9692	0.9940	0.8171	0.932
		5	10	0.9761	0.9964	0.8291	0.9415
		5	20	0.9829	0.9971	0.8530	0.956
		10	10	0.9880	0.9978	0.8769	0.967
		10	20	0.9987	0.9976	0.9411	0.981
	0.5	1	10	0.9551	0.9865	0.8085	0.9151
		1	20	0.9675	0.9935	0.7949	0.9308
		5	10	0.9744	0.9961	0.8308	0.94
		5	20	0.9778	0.9965	0.8496	0.955
		10	10	0.9829	0.9982	0.8821	0.966
		10	20	0.9915	0.9975	0.9316	0.9835
	0.3	1	10	0.9470	0.9894	0.8103	0.9237
		1	20	0.9709	0.9937	0.8137	0.929
		5	10	0.9744	0.9969	0.8410	0.9465
		5	20	0.9795	0.9971	0.8581	0.953
		10	10	0.9863	0.9977	0.8855	0.966
		10	20	0.9915	0.9976	0.9385	0.9825
Breast Ultrasound Dataset	0.9	1	10	0.8077	0.8558	0.6538	0.800
		1	20	0.8462	0.9090	0.6792	0.86
		5	10	0.9615	0.9985	0.8205	0.957
		5	20	0.9872	0.9993	0.8974	0.971
		10	10	0.9872	0.9978	0.8846	0.9613
		10	20	0.9872	1.0000	0.8974	0.946
	0.5	1	10	0.8462	0.8439	0.6282	0.773
		1	20	0.8205	0.8609	0.6795	0.8463
		5	10	0.8590	0.9364	0.7692	0.914
		5	20	0.9744	0.9919	0.8846	0.922
		10	10	0.9872	1.0000	0.8846	0.931
		10	20	1.000	1.000	0.9231	0.96
	0.3	1	10	0.6282	0.8410	0.5641	0.857
		1	20	0.8642	0.9016	0.6667	0.894
		5	10	0.9231	0.9845	0.8974	0.922
		5	20	1.0000	1.0000	0.8846	0.95
		10	10	0.9744	0.9970	0.8846	0.934
		10	20	0.9872	1.0000	0.9231	0.968

It is observed that FMT-SVM achieves consistently strong performance across all configurations. Notably, performance improves steadily with increasing local epochs (E) and communication rounds (R), indicating stable convergence behavior.

Importantly, these trends are consistent across both datasets, despite significant differences in modality (chest X-ray vs. ultrasound), dataset size, and noise characteristics. This consistency across diverse experimental settings highlights the robustness and generalization capability of the proposed method in real-world healthcare scenarios.

3.3.1. Effect of Data Heterogeneity ($\alpha$)

This experiment directly evaluates the robustness of the proposed framework under different levels of statistical heterogeneity. Figure 5 and Figure 6 illustrate the accuracy obtained across all E and R settings for $\alpha =0.9$, $0.5$, and $0.3$. It is observed that lower $\alpha$ values (higher heterogeneity) generally reduce performance, particularly in Task 2 for both datasets. In the pneumonia dataset, at $E=5$ and $R=20$, Task 2 accuracy decreases from 85.30% ($\alpha =0.9$) to 84.96%, and when E is small, i.e $E=1$, accuracy decreases from $81.71$ to $81.37\%$. For the breast ultrasound dataset, data heterogeneity effects are more visible in the binary Task 1, where at $E=1$ and $R=10$, the accuracy drops from $80.77\%$ ($\alpha =0.9)$ to 62.82% $(\alpha =0.3)$. Additionally, per-class AUC values show that minority classes (class 1 in the pneumonia dataset and class 0 in the breast cancer dataset) are most impacted by non-IID data, while the majority classes maintain high separability, even at low $\alpha$ values.

Figure 5. Accuracy curve for the Pneumonia Dataset for E $\in \{10,20\}$ with $\alpha \in \{0.9,0.5,0.3\}$ for 10 and 20 global rounds. (a) 10 global rounds for $\alpha =0.9$; (b) 20 global rounds for $\alpha =0.9$; (c) 10 global Rounds for $\alpha =0.5$; (d) 20 global rounds for $\alpha =0.5$; (e) 10 global rounds for $\alpha =0.3$; (f) 20 global rounds for $\alpha =0.3$.

Figure 6. Accuracy curve for the Breast Ultrasound Dataset for E $\in \{10,20\}$ with $\alpha \in \{0.9,0.5,0.3\}$ for 10 and 20 global rounds. (a) 10 global rounds for $\alpha =0.9$; (b) 20 global rounds for $\alpha =0.9$; (c) 10 global rounds for $\alpha =0.5$; (d) 20 global rounds for $\alpha =0.5$; (e) 10 global rounds for $\alpha =0.3$; (f) 20 global rounds for $\alpha =0.3$.

3.3.2. Impact of Local Epochs (E)

We investigated the effect of local training iterations by varying the number of local epochs ($E\in \{1,5,10\}$). It is observed that increasing the number of local epochs leads to substantial performance boosts, particularly under high heterogeneity. For the Pediatric Pneumonia Dataset with $\alpha =0.3$, increasing E from 1 to 10 increases Task 1 ACC from 94.70% to 98.63% and Task 2 ACC from 81.03% to 88.55% at $R=10$ and Task 1 ACC from 97.09% to 99.15% and Task 2 ACC from 81.37% to 93.85% at $R=20$. Similarly, for the Breast Ultrasound Dataset at $\alpha =0.3$, increasing E from 1 to 10 increases Task 1 ACC from 62.82% to 97.44% and Task 2 ACC from 56.41% to 88.46% at $R=10$ and Task 1 ACC from 86.42% to 98.72% and Task 2 ACC from 66.67% to 92.31% at $R=20$. This highlights the importance of allowing clients to perform deeper local updates before aggregation, especially in smaller and noisier datasets.

3.3.3. Effect of Global Epochs (R)

We evaluated the impact of communication rounds by comparing $R\in \{10,20\}$. It is observed that, across both datasets, increasing the number of global rounds (R) consistently improves model performance, especially for more heterogeneous data (i.e., $\alpha =0.3$). For the Pediatric Pneumonia Dataset, Task 1 ACC rises from 94.70% at R = 10 to 97.09% at R = 20 ($\alpha =0.3,E=1$). For Task 2, when the E value is large (i.e., E = 10), the effect of increasing R is more significant; when $\alpha$ = 0.3, the accuracy increases from 88.55% to 93.85%, and when $\alpha =0.5$, it increases from 88.21% to 93.16%. A similar trend is observed for the Breast Ultrasound Dataset; when $\alpha =0.3$, for Task 1, accuracy increases from 62.82% at R = 10 to 86.42% at R = 20. Hence, the number of global rounds (R) plays a stabilizing role, especially under challenging non-IID conditions, helping to mitigate the drift caused by local updates, in addition to compensating for data heterogeneity.

3.3.4. Task-Specific Sensitivity

Task 2 (multi-class classification) exhibits greater sensitivity to all three parameters ($\alpha$, R, and E) than Task 1 (binary classification). This highlights the need for careful calibration of federated training configurations based on task complexity and class distribution across clients.

3.4. Comparison of FMT-SVM with FedPureCNN and FedPureSVM

Table 6 presents the empirical performance and communication cost of FMT-SVM compared with FedPureCNN and FedPureSVM across both datasets. We assessed the performance of the three models based on accuracy and communication cost. It is observed that FMT-SVM consistently outperforms FedPureSVM and FedPureCNN across both datasets and configurations. Furthermore, since FMT-SVM transmits only the SVM head parameters, it has a per-round communication cost of a few kilobytes.

Table 6. Comparison of accuracy and communication cost for FMT-SVM, FedPureCNN, and FedPureSVM.

Dataset	Method	E	Task1Acc	Task2Acc	Comm. Cost per Round
Pediatric Pneumonia Dataset	FMT-SVM	5	97.78%	84.96%	2.06 KB
		10	99.15%	93.16%	2.06 KB
	FedPureCNN	5	91.11%	75.21%	4.02 MB
		10	92.99%	73.68%	4.02 MB
	FedPureSVM	5	91.97%	66.32%	0.5 KB
		10	92.14%	71.67%	0.5 KB
Breast Ultrasound Dataset	FMT-SVM	5	97.44%	88.46%	2.06 KB
		10	100%	92.31%	2.06 KB
	FedPureCNN	5	82.05%	65.38%	4.02 MB
		10	82.05%	66.67%	4.02 MB
	FedPureSVM	5	69.12%	57.61%	0.5 KB
		10	73.92%	65.23%	0.5 KB

Quantitatively, for a feature dimension of $d=128$ and $C=3$ classes, FMT-SVM requires only $2.06$ KB per client per round, whereas FedPureCNN requires approximately $4.02$ MB per round, resulting in a communication reduction of nearly $400\times$. While FedPureSVM has a comparable communication cost (≈0.5 KB per round), it exhibits substantially lower accuracy due to the absence of deep feature representations.

Considering multiple communication rounds, this reduction becomes even more significant. For example, over 10 rounds, FedPureCNN incurs approximately $40.2$ MB of communication cost per client, whereas FMT-SVM requires only ∼20 KB.

Overall, FedPureCNN achieves reasonable accuracy but at the expense of extremely high communication cost, while FedPureSVM is communication-efficient but lacks predictive performance. In contrast, FMT-SVM effectively balances both aspects by leveraging deep feature representations with minimal communication overhead.

3.5. Privacy Enhancement of FMT-SVM with Differential Privacy

To enhance privacy in the proposed FMT-SVM framework, we incorporate Gaussian-based differential privacy (DP) into the client-side updates. Although FL avoids sharing raw data, it remains vulnerable to privacy risks such as gradient leakage and model inversion attacks, which can potentially infer sensitive information from shared updates. To mitigate these risks, we apply DP to the communicated parameters.

In our framework, DP is applied specifically to the client-side SVM head updates before transmission to the server. After local training, the SVM head parameters $(w_1,W_2)$ are first clipped using an ${\mathcal{l}}_2$ norm bound clip_norm to limit sensitivity. Gaussian noise ($\mathcal{N}(0,{\sigma }^2)$) is then added to the clipped parameters. The encoder parameters (${\theta }_k$) remain local to each client and are neither shared nor perturbed, thereby preserving local feature representations while reducing the exposure of sensitive information.

The noise scale ($\sigma$) is computed as follows:

$$\sigma ={\displaystyle \frac{\mathrm{clip}\_\mathrm{norm}·\sqrt{2\log (1.25/\delta )}}{ϵ}},$$

(19)

where $ϵ\in \{0.8,1.0\}$ is the privacy budget, $\delta ={10}^{-5}$, and $\mathrm{clip}\_\mathrm{norm}=1.0$. Using Equation (19), the noise scale is $\sigma =4.884$ for $ϵ=1.0$ and $\sigma =6.05$ for $ϵ=0.8$.

Table 7 presents the classification accuracies of FMT-SVM on the Pediatric Pneumonia and Breast Ultrasound datasets under varying local epochs (E) and privacy budgets. We compare performance without DP ($ϵ=0$) against moderate privacy settings ($ϵ=0.8$ and $1.0$).

Table 7. Comparison of FMT-SVM with DP and without DP.

Dataset	Method	E	$\mathit{ϵ}$	Task1Acc	Task2Acc
Pediatric Pneumonia Dataset	FMT-SVM without DP	5	0.0	97.78%	84.96%
		10		99.15%	93.16%
	FMT-SVM with DP	5	0.8	93.2%	77.89%
		10		82.56%	72.48%
		5	1.0	94.78%	79.96%
		10		85.78%	78.19%
Breast Ultrasound Dataset	FMT-SVM without DP	5	0.0	97.44%	88.46%
		10		100.0%	92.31%
	FMT-SVM with DP	5	0.8	94.12%	89.81%
		10		93.62%	85.58%
		5	1.0	99.15%	91.87%
		10		97.71%	91.34%

It is observed that in FMT-SVM with DP, the model experiences a reduction in accuracy due to the injected noise, reflecting the inherent privacy–utility trade-off. At a stricter privacy level ($ϵ=0.8$), performance degradation is more pronounced, particularly for the Pediatric Pneumonia Dataset, where Task 1 accuracy decreases from 97.78% to 93.20% (5 epochs) and further to 82.56% (10 epochs). Interestingly, on the Breast Ultrasound Dataset, FMT-SVM maintains relatively high accuracies under DP, achieving up to 94.12% and 93.62% on Task 1 for 5 and 10 epochs, respectively, suggesting robustness to privacy noise in this domain.

At a slightly relaxed privacy budget of $ϵ$ = 1.0, performance recovers somewhat. For instance, Task 1 accuracy on the Pediatric Pneumonia Dataset improves to 94.78% and 85.78% for 5 and 10 epochs, respectively, while Task 2 accuracy reaches up to 79.96% and 78.19%. Similarly, on the Breast Ultrasound Dataset, Task 1 accuracy remains strong, at 99.15% (5 epochs) and 97.71% (10 epochs). These results indicate that FMT-SVM can effectively balance privacy and utility under practical DP configurations.

Overall, by applying noise only to the communicated SVM head parameters, the proposed framework reduces the risk of information leakage while maintaining strong predictive performance, making it suitable for privacy-sensitive federated healthcare applications.

4. Discussion

The above experimental results show the proposed FMT-SVM framework achieves better accuracy for a variety of healthcare tasks and is communication-efficient. In this section, we first discuss the impact of various regularization terms on the performance of the FMT-SVM framework, then compare its performance with that of existing approaches in SVM-based federated learning (single-task), personalized FL, and MTFL. Our comparison focuses not on raw accuracy, since it is highly dataset-dependent, but on methodological design, computational efficiency, scalability, and practical applicability in real-world federated systems.

4.1. Ablation Study

To comprehend the sensitivity and contribution of each component in our FMT-SVM framework, we conducted ablation experiments by varying the regularization hyperparameters:

• $\lambda$: Task covariance regularization weight ($\mathrm{Tr}\left(W^{\top }{\mathsf{\Sigma }}^{-1}W\right)$);
• $\mu$: Global consistency regularization weight $({\parallel W-W_{\mathrm{global}}\parallel }^2$);
• C: SVM margin penalty in the hinge loss.

We vary each hyperparameter while keeping the others fixed and present the binary and multi-class classification accuracies for the Pediatric Pneumonia Dataset with $\alpha =0.5$.

4.1.1. Effect of $\lambda$: Task Covariance Regularization

We assess the strength of task covariance regularization ($\lambda$), which governs the sharing of statistical strength across binary and multi-class classification tasks. Figure 7 and Table 8 present the ablation study on task covariance regularization ($\lambda$, with fixed $C=1.0$ and $\mu =0.01$). It is observed that without this regularization (i.e., when $\lambda =0$), due to the absence of task coupling, the result is weaker knowledge transfer, with average binary accuracy plateauing around $94.1\%$ and multi-class accuracy around $87.7\%$. Introducing a mild regularization ($\lambda =0.01$) boosts binary accuracy to $98.9\%$, although multi-class performance fluctuates (average $\approx 83.3\%$) due to insufficient structural sharing. Furthermore, increasing $\lambda$ to $0.1$ produces the best overall results, achieving a peak binary accuracy of $99.15\%$ and multi-class accuracy of $95.38\%$, showing strong positive transfer between tasks. However, further increasing $\lambda$ to $1.0$ introduces over-smoothing, leading to degraded performance causing binary accuracy to drop to ≈97.26% and multi-class accuracy to $82.56\%$. This shows the need for a balanced degree of task-level coupling in multi-task federated SVMs.

Figure 7. Task covariance regularization ($\lambda$, with fixed $C=1.0$ and $\mu =0.01$).

Table 8. Ablation study on task covariance regularization ($\lambda$, with fixed $C=1.0$ and $\mu =0.01$) in terms of accuracy.

$\mathit{\lambda }$	Binary	Multi-Class	Observations
0.0	0.9415	0.8957	No task coupling, leading to weaker generalization
0.01	0.9897	0.9179	Moderate gain from task-sharing regularization
0.1	0.9910	0.9521	Optimal coupling across tasks
1.0	0.9726	0.7880	Over-regularization degrades both tasks

4.1.2. Effect of $\mu$: Global Consistency Regularization

We assess the influence of the global consistency regularization term ($\mu$), which enforces the proximity of client models to the global model. Figure 8 and Table 9 present the ablation study on global consistency regularization ($\mu$, with fixed $C=1.0$ and $\lambda =0.1$). It is observed that when $\mu =0$, local models diverge significantly, leading to generalization across clients. A small $\mu =0.01$ achieves the best trade-off between personalization and consensus, attaining a top binary accuracy of $99.14\%$ and multi-class accuracy of $95.24\%$. As $\mu$ increases beyond this, the model becomes too rigid, limiting how much each client can adapt locally; hence, there is a drop in overall performance. Therefore, these results show the importance of the critical trade-off between global model consistency and client adaptability, hence, achieving efficient personalization without compromising model convergence.

Figure 8. Global consistency regularization ($\mu$, with fixed $C=1.0$ and $\lambda =0.1$).

Table 9. Ablation study on global consistency regularization ($\mu$, with fixed $C=1.0$ and $\lambda =0.1$) in terms of accuracy.

$\mathit{\mu }$	Binary	Multi-Class	Observations
0.0	0.9914	0.8581	No consensus; hence, client overfit to local bias
0.01	0.9914	0.9524	Promotes global model consistency
0.1	0.9914	0.9163	Slight degradation due to over-constraining
1.0	0.9880	0.8119	Strong penalization hinders personalization

4.1.3. Effect of C: SVM Margin Penalty

We assess the impact of the penalty parameter (C), which balances the trade-off between maximizing the margin and penalizing hinge-loss violations in the primal formulation. Table 10 presents the ablation study on SVM margin penalty (C, with fixed $\lambda =0.1$ and $\mu =0.01$), and Figure 9 presents the convergence of the model with these settings. It is observed that when $C=0.01$, the result is poor performance, likely due to under-regularization, allowing for large slack violations, which leads to a weak decision boundary. As C increases to $0.1$, performance improves significantly across all metrics, striking a good balance between margin maximization and misclassification penalty. At $C=1.0$, we achieve the best results, with binary accuracy reaching $98.9\%$ and multi-class accuracy reaching $92.7\%$, providing optimal control over margin width and error penalization, yielding both stable and discriminative classifiers. However, when C is increased further to $10.0$, performance degrades badly. This is attributed to overfitting to local client data and possible instability during global aggregation, where excessive penalties on hinge loss introduce large gradients that disrupt convergence. These results highlight the sensitivity of C and suggest that moderate values (viz., $C=1.0$) yield the most reliable performance in non-IID federated settings.

Table 10. Ablation study on penalty parameter (C, with fixed $\lambda$ and $\mu$).

C	Binary	Multi-Class	Observations
0.01	0.8752	0.6222	Underfitting due to an excessively soft margin and weak decision boundaries
0.1	0.9555	0.7743	Unstable performance
1.0	0.9897	0.9271	Best performance overall
10.0	0.7470	0.4615	Severe overfitting or divergence

Figure 9. SVM margin penalty (C, with fixed $\lambda =0.1$ and $\mu =0.01$).

4.2. Comparison with SVM-Based Federated Learning (Single-Task)

Early federated SVM methods largely focused on training a single global model for both IID and non-IID settings but failed to capture task-specific variations in heterogeneous environments. Table 11 compares our FMT-SVM with key single-task SVM-based federated learning approaches, highlighting differences in methodological design, communication efficiency, scalability, and privacy features. Our FMT-SVM employs a primal formulation with task covariance regularization, communicating only compact SVM heads. This design enables efficient, scalable, and privacy-aware MTFL across heterogeneous clients. In contrast, Fed-KSVM relies on primal parameter sharing with moderate communication costs. SVF reduces communication by transmitting sparse support vectors and optionally incorporates differential privacy. BDSVM exchanges kernel matrices, which can increase communication overhead. SecretSVM offers strong privacy guarantees through secure multiparty computation but at the cost of higher communication and computational demands. TurboSVM-FL achieves scalability via class-embedding aggregation but lacks formal privacy protections.

Table 11. Comparison of single-task SVM-based federated learning methods in terms of design, communication efficiency, scalability, and privacy/security.

Method	Methodological Design	Communication Efficiency	Scalability	Privacy/Security
Fed-KSVM [31]	Parameter sharing in primal domain	Moderate; exchanges full model parameters	Moderate scalability; sensitive to model size	No formal privacy guarantees
SVF [33]	Perturbed support vectors in dual domain	Efficient if support vectors are sparse	Moderate; depends on SV count	Supports optional differential privacy via noise addition
BDSVM [30]	Kernel Gram matrix exchange (function space)	Low communication efficiency; kernel matrices are large	Poor scalability for large datasets	No privacy protection
SecretSVM [32]	Secure multiparty computation with encrypted updates	High communication and computation overhead	Limited scalability due to cryptographic complexity	Strong privacy via secret sharing (SMPC)
TurboSVM-FL [34]	Aggregates class embeddings instead of parameters	Low communication overhead	Scalable for moderate numbers of classes	No privacy guarantees
FMT-SVM (Ours)	Primal–dual formulation with task covariance regularization	Very efficient; communicates only compact SVM heads (not entire model)	Highly scalable; handles heterogeneous tasks efficiently	Optional differential privacy with noise calibration

4.3. Comparison with Personalized FL and Multi-Task FL

Personalized FL methods such as FedPer, FedRep, CPFL, and FedBABU handle data heterogeneity by learning a shared global encoder among clients while each client maintains local classifier heads. These methods are effective in non-IID data distributions, but they are restricted to solving a single global task and cannot accommodate multiple heterogeneous tasks within the same client; hence, they are better suited for personalized learning rather than true multi-task learning.

In contrast, state-of-the-art MTFL approaches such as MOCHA consider each client’s data heterogeneity as an independent task, which is different from true multi-task learning in existing healthcare applications. Moreover, MOCHA suffers from high communication overhead because of its iterative convex optimization, which limits its scalability.

Furthermore, deep MTFL methods such as FedBone and MTL use shared deep backbones with task-specific heads or personalized layers. These approaches support multi-task learning but remain neural network-centric, involving large model updates, heavy computational overhead, and limited interpretability.

Table 12 summarizes notable MTFL methods in the literature. Unlike many existing MTFL methods that focus primarily on personalization or deep multi-task architectures, our FMT-SVM framework offers a distinct way of handling heterogeneity. It provides true multi-task modeling by jointly supporting multiple task types (e.g., binary and multi-class) within each client using lightweight SVM heads coupled with a unified local encoder. Unlike personalization approaches, it does not restrict clients to a single task. Unlike MOCHA, it enables joint multi-task optimization instead of treating clients as independent tasks. Compared to deep MTFL methods, it remains lightweight, interpretable, and communication-efficient by transmitting only compact SVM heads rather than entire deep networks. Furthermore, by combining classical SVM theory with a CNN encoder and primal–dual optimization, FMT-SVM achieves efficient parameter sharing while maintaining compatibility with privacy-preserving mechanisms.

Table 12. Comparison of MTFL approaches based on design, communication efficiency, scalability, and privacy/security.

Method	Methodological Design	Communication Efficiency	Scalability	Privacy/Security
MOCHA [35]	Convex models with per-client tasks treated independently (personalization)	Low to moderate; high communication cost due to convex optimization	Moderate scalability; robust to stragglers and faults	Moderate; no explicit privacy mechanisms
VIRTUAL [36]	Bayesian neural networks with latent task mixture	Medium communication cost	Moderate scalability	Limited privacy guarantees
MTFL [37]	Deep networks with personalized layers	Medium communication cost	High scalability	Limited privacy guarantees
FedBone [42]	Deep networks with shared backbone and task heads	Medium communication cost	High scalability	Limited privacy guarantees
MTL [43]	End-to-end neural nets, shared task reps, and no classical ML integration	Moderate; full model transfer	Scales but sensitive to heterogeneity	Secure aggregation; no raw data sharing
FMT-SVM (Ours)	SVMs + CNN encoder with primal multi-task optimization	Low to medium; only compact SVM heads communicated	High scalability; effective for heterogeneous clients	Optional differential privacy via Gaussian noise addition

From a practical deployment perspective, FMT-SVM is well suited for real-world healthcare settings such as hospitals, diagnostic centers, and edge medical devices, where data is inherently distributed, heterogeneous, and privacy-sensitive. Since the feature encoder remains local and only lightweight SVM parameters are communicated, the framework significantly reduces bandwidth requirements and is suitable for deployment in resource-constrained environments.

Additionally, the incorporation of task covariance regularization enhances robustness to data heterogeneity by effectively capturing inter-task relationships. The federated learning paradigm further ensures data privacy by avoiding centralized data sharing. The framework is also resilient to variations in client data distributions and partial client participation, making it suitable for dynamic and large-scale federated networks.

Overall, FMT-SVM bridges the gap between personalized federated learning and multi-task federated learning, offering a practical, robust, and privacy-aware solution for multi-disease prediction in real-world healthcare applications, where heterogeneity is inherently multi-dimensional.

5. Conclusions and Future Work

In this work, we proposed a novel federated multi-task learning framework, FMT-SVM, that integrates a unified CNN encoder with task-specific linear SVM heads optimized via a primal formulation. Our approach explicitly models task relationships within each client through covariance regularization while enabling effective knowledge sharing across heterogeneous clients with different data distributions and tasks.

A major strength of the proposed FMT-SVM framework over peers is its communication efficiency. Unlike traditional FL methods that require the exchange of entire model parameters or large SV sets, FMT-SVM only communicates the compact SVM heads during aggregation. This significantly reduces communication overhead, also making the method highly scalable and suitable for deployment in real-world federated systems with bandwidth constraints and many clients. Furthermore, FMT-SVM ensures enhanced privacy protection by incorporating Gaussian DP mechanisms inclient updates, achieving a practical balance between privacy guarantees and predictive performance.

Extensive experiments on the Pediatric Pneumonia Dataset and the Breast Ultrasound Dataset demonstrate that FMT-SVM achieves competitive accuracy while maintaining communication and privacy advantages over baseline single-task and MTFL methods.

Future work will explore the extension of the framework to support nonlinear kernels and adaptive task covariance structures, as well as real-world deployment scenarios involving asynchronous updates and heterogeneous client capabilities.