As data governance and privacy concerns increasingly limit centralized data collection, learning from decentralized datasets has become essential. Federated Learning (FL) provides a principled framework for this setting by enabling collaborative training across decentralized data silos. However, deploying FL in practice remains challenging due to communication costs, system heterogeneity, robustness and privacy requirements, and the need to support the full data lifecycle, including unlearning. Many of these challenges are further compounded by the inherently heterogeneous nature of client data. This thesis studies the design of practical federated learning systems under data heterogeneity through a combination of algorithmic and system-level techniques.
First, the thesis addresses communication efficiency and data heterogeneity via novel ensemble schemes in the one-shot FL paradigm, where communication is restricted to a single round. In particular, it shows that training prediction aggregators over client ensembles can approach the accuracy of iterative FL while maintaining communication costs comparable to one-shot FL. The thesis further introduces flexible model merging techniques that reduce the ensemble size for practical deployment, revealing favorable accuracy-size trade-offs, and develops robust aggregators that enable federated ensembles to remain accurate in the presence of adversarial or faulty clients.
Second, the thesis addresses the problem of federated unlearning. It shows that in situ generation of distilled datasets during federated training enables fast and scalable unlearning without retraining from scratch or storing raw data or gradients, supporting multiple deletion requests with orders-of-magnitude efficiency gains.
Third, the thesis revisits privacy assumptions in collaborative learning by exploring system-level support through trusted execution environments. It demonstrates that limited raw data sharing within hardware-enforced enclaves can significantly improve training efficiency and convergence while preserving strong privacy guarantees.
Lastly, the thesis tackles system heterogeneity in federated deployments. It introduces cohort-parallel training to better harness large client participation and gradient-free guessed updates to mitigate the impact of stragglers, substantially improving convergence speed and resource efficiency.
Collectively, these contributions advance the state of practical FL by reconciling accuracy, communication efficiency, robustness, privacy, and scalability under realistic data and system constraints. The thesis concludes by outlining broader design principles and future directions for federated learning systems as they increasingly intersect with large-scale and continually evolving AI models.
