Layer-Level Knowledge Distillation for Deep Neural Network Learning
AbstractMotivated by the recently developed distillation approaches that aim to obtain small and fast-to-execute models, in this paper a novel Layer Selectivity Learning (LSL) framework is proposed for learning deep models. We firstly use an asymmetric dual-model learning framework, called Auxiliary Structure Learning (ASL), to train a small model with the help of a larger and well-trained model. Then, the intermediate layer selection scheme, called the Layer Selectivity Procedure (LSP), is exploited to determine the corresponding intermediate layers of source and target models. The LSP is achieved by two novel matrices, the layered inter-class Gram matrix and the inter-layered Gram matrix, to evaluate the diversity and discrimination of feature maps. The experimental results, demonstrated using three publicly available datasets, present the superior performance of model training using the LSL deep model learning framework. View Full-Text
Share & Cite This Article
Li, H.-T.; Lin, S.-C.; Chen, C.-Y.; Chiang, C.-K. Layer-Level Knowledge Distillation for Deep Neural Network Learning. Appl. Sci. 2019, 9, 1966.
Li H-T, Lin S-C, Chen C-Y, Chiang C-K. Layer-Level Knowledge Distillation for Deep Neural Network Learning. Applied Sciences. 2019; 9(10):1966.Chicago/Turabian Style
Li, Hao-Ting; Lin, Shih-Chieh; Chen, Cheng-Yeh; Chiang, Chen-Kuo. 2019. "Layer-Level Knowledge Distillation for Deep Neural Network Learning." Appl. Sci. 9, no. 10: 1966.
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.