Knowledge Distillation in Object Detection: A Survey from CNN to Transformer

Deep learning models, especially for object detection have gained immense popularity in
computer vision. These models have demonstrated remarkable accuracy and performance,
driving advancements across various applications. However, the high computational
complexity and large storage requirements of state-of-the-art object detection models pose
significant challenges for deployment on resource-constrained devices like mobile phones
and embedded systems. Knowledge Distillation (KD) has emerged as a prominent solution
to these challenges, effectively compressing large, complex teacher models into smaller,
efficient student models. This technique maintains good accuracy while significantly
reducing model size and computational demands, making object detection models more
practical for real-world applications. This survey provides a comprehensive review of KDbased
object detection models developed in recent years. It offers an in-depth analysis of
existing techniques, highlighting their novelty and limitations, and explores future research
directions. The survey covers the different distillation algorithms used in object detection.
It also examines extended applications of knowledge distillation in object detection, such
as improvements for lightweight models, addressing catastrophic forgetting in incremental
learning, and enhancing small object detection. Furthermore, the survey also delves into
the application of knowledge distillation in other domains such as image classification,
semantic segmentation, 3D reconstruction, and document analysis.