상세 보기
Sparse MoE Students for Efficient Knowledge Distillation
- Ryu, Jongwon;
- Kim, Mingi;
- Kim, Junyeong
WEB OF SCIENCE
1SCOPUS
2초록
We present a sparse Mixture-of-Experts student architecture for knowledge distillation that replaces conventional dense students with a modular design of lightweight, class-agnostic experts dynamically routed by input-conditioned gating. Our approach provides three key benefits: (i) accuracy–efficiency Pareto superiority over dense students by activating only a subset of experts per input, (ii) loss-agnostic plug-and-play compatibility, consistently improving performance across diverse Knowledge distillation objectives, and (iii) class-agnostic expert decomposition, enabling generalization without label supervision. To ensure fairness, we construct both dense and Mixture-of-Experts students with comparable parameter budgets and systematically evaluate multiple gating strategies, including soft, top-k, and attention-based variants. Experiments on CIFAR-10 and CIFAR-100 under four distillation methods show that attention-gated MoE students consistently outperform dense baselines at equal capacity, achieving up to +1.7% accuracy gains with an average 25% FLOPs reduction. These findings demonstrate the structural advantages of sparse Mixture-of-Experts students as efficient, generalizable, and loss-agnostic alternatives to dense models in knowledge distillation.
키워드
- 제목
- Sparse MoE Students for Efficient Knowledge Distillation
- 저자
- Ryu, Jongwon; Kim, Mingi; Kim, Junyeong
- 발행일
- 2025
- 유형
- Article
- 저널명
- IEEE Access
- 권
- 13
- 페이지
- 187373 ~ 187382
- 언어
- ENG
- 출판사
- Institute of Electrical and Electronics Engineers Inc.
- 발행국가
- 미국
- 분량
- 10 페이지
- ISSN
- E 2169-3536
P 2169-3536