Federated Knowledge Transfer for Heterogeneous Visual Models

Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables collaborative training of machine learning models among multiple participants. However, despite recent progress, existing federated learning systems can still not handle heterogeneous models. For instance, candidate clients with heterogeneous models are inaccessible to the established federated system. And within the federated system, local models are forbidden to be updated to become heterogeneous models, even though the updated models work better. Considering the reality of heterogeneous models, we study two practical scenarios, Local Model Update Scenario and Hetero-Model Enrollment Scenario. We then proposes a novel method to tackle the problems, which we refer to as Federated learning with deep-layer Feature Alignment (FedDFA). FedDFA uses deep-layer knowledge distillation to align the feature representation and solve the knowledge transfer problem of heterogeneous models. We constructed a federated learning system where we take convolutional neural networks (CNNs) as local models and vision transformers (ViT) as heterogeneous models. We trained these models with three datasets (CIFAR-10, CELEBA, and ImageNet-1k) and their non-I.I.D. variants. As a result, our approach facilitates FL with wide applicability for various models and better generalization performance than the state-of-the-art methods.