Sparse MoE Students for Efficient Knowledge Distillation

Citations

WEB OF SCIENCE

1
Citations

SCOPUS

2

초록

We present a sparse Mixture-of-Experts student architecture for knowledge distillation that replaces conventional dense students with a modular design of lightweight, class-agnostic experts dynamically routed by input-conditioned gating. Our approach provides three key benefits: (i) accuracy–efficiency Pareto superiority over dense students by activating only a subset of experts per input, (ii) loss-agnostic plug-and-play compatibility, consistently improving performance across diverse Knowledge distillation objectives, and (iii) class-agnostic expert decomposition, enabling generalization without label supervision. To ensure fairness, we construct both dense and Mixture-of-Experts students with comparable parameter budgets and systematically evaluate multiple gating strategies, including soft, top-k, and attention-based variants. Experiments on CIFAR-10 and CIFAR-100 under four distillation methods show that attention-gated MoE students consistently outperform dense baselines at equal capacity, achieving up to +1.7% accuracy gains with an average 25% FLOPs reduction. These findings demonstrate the structural advantages of sparse Mixture-of-Experts students as efficient, generalizable, and loss-agnostic alternatives to dense models in knowledge distillation.

키워드

Knowledge distillation; mixture-of-experts (MoE); routing strategy; sparse models
제목
Sparse MoE Students for Efficient Knowledge Distillation
저자
Ryu, Jongwon; Kim, Mingi; Kim, Junyeong
DOI
10.1109/ACCESS.2025.3623953
발행일
2025
유형
Article
저널명
IEEE Access
권
13
페이지
187373 ~ 187382

파일 다운로드

Thumbnail