Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
4.9
2.1
Journals
Entropy
Special Issues
Information Theory and Deep Neural Networks
entropy-logo
Submit to Entropy Review for Entropy Propose a Special Issue

Journal Menu

► Journal Menu

Journal Browser

► Journal Browser
share announcement

Information Theory and Deep Neural Networks

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Information Theory, Probability and Statistics".

Deadline for manuscript submissions: closed (30 November 2021) | Viewed by 15390

Special Issue Editors

Prof. Dr. Shu-Tao Xia

E-Mail Website
Guest Editor
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
Interests: computer vision; information theory; algebraic coding theory; machine learning; deep learning; internet; distributed storage
Special Issues, Collections and Topics in MDPI journals
Dr. Bin Chen

E-Mail Website
Guest Editor
Department of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, China
Interests: information theory; data compression; algebraic coding theory; machine learning; deep learning; distributed storage
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Information theory answers two fundamental questions in communication theory, but its impact has expanded far beyond the field of communication. Many methods and ideas developed in information theory have been adopted to explain and uncover the internal mechanism in modern deep neural networks. For example, the information bottleneck has served as a tool in explaining the representation learning; mutual information has been adopted in deriving various types of generalization bounds in a deep neural network; quantization and other data compression techniques have been applied in the design of lightweight deep neural networks, namely model compression. Progress regarding an information theoretic understanding of deep neural networks has often been driven by the deep-learning-based application and induced phenomenon and is yet to be explored further.

Neural networks have a long history, aiming to understand how the human brain works and how what we call intelligence is formed. Neural networks with many layers, known as deep neural networks (DNNs), encompassing convolutional neural networks (CNN) and recurrent neural networks (RNN), have become popular and achieved state-of-the-art performance in various computer vision tasks. However, the use of deep neural networks to study and improve the classical source coding and channel coding problem in information theory is also yet to be explored. There have been some advances in applying DNNs to compressed sensing, image and video compression, channel decoding, and joint source-channel coding.   

This Special Issue aims to provide an opportunity for the presentation of novel progress regarding the intersection between information theory and deep neural networks. Specifically, the information theoretic analysis and interpretation of DNN-based applications and induced phenomena, in addition to the design of improved coding schemes by DNNs in signal processing, data compression, channel coding or other topics in information theory, fall within the scope of this Special Issue. Contributions addressing any of these issues are very welcome.

Prof. Dr. Shu-Tao Xia
Dr. Bin Chen
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • information theoretic techniques
  • statistics
  • deep learning
  • model compression
  • data compression
  • compressed sensing
  • optimization
  • generalization
  • generative adversarial networks
  • graph convolutional networks
  • joint source-channel coding
  • applications

Benefits of Publishing in a Special Issue

  • Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
  • Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
  • Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
  • External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
  • e-Book format: Special Issues with more than 10 articles can be published as dedicated e-books, ensuring wide and rapid dissemination.

Further information on MDPI's Special Issue policies can be found here.

Published Papers (4 papers)

Download All Papers
Order results
Result details
Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:

Research

27 pages, 12303 KiB  
Article
Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
by Baihan Lin
Entropy 2022, 24(1), 59; https://doi.org/10.3390/e24010059 - 28 Dec 2021
Cited by 3 | Viewed by 2814
Abstract
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network [...] Read more.
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network optimization process as a partially observable model selection problem, the regularity normalization constrains the implicit space by a normalization factor, the universal code length. We compute this universal code incrementally across neural network layers and demonstrate the flexibility to include data priors such as top-down attention and other oracle information. Empirically, our approach outperforms existing normalization methods in tackling limited, imbalanced and non-stationary input distribution in image classification, classic control, procedurally-generated reinforcement learning, generative modeling, handwriting generation and question answering tasks with various neural network architectures. Lastly, the unsupervised attention mechanisms is a useful probing tool for neural networks by tracking the dependency and critical learning stages across layers and recurrent time steps of deep networks. Full article
(This article belongs to the Special Issue Information Theory and Deep Neural Networks)
►▼ Show Figures

Figure 1

15 pages, 1428 KiB  
Article
SAMLDroid: A Static Taint Analysis and Machine Learning Combined High-Accuracy Method for Identifying Android Apps with Location Privacy Leakage Risks
by Guangwu Hu, Bin Zhang, Xi Xiao, Weizhe Zhang, Long Liao, Ying Zhou and Xia Yan
Entropy 2021, 23(11), 1489; https://doi.org/10.3390/e23111489 - 10 Nov 2021
Cited by 7 | Viewed by 3036
Abstract
Insecure applications (apps) are increasingly used to steal users’ location information for illegal purposes, which has aroused great concern in recent years. Although the existing methods, i.e., static and dynamic taint analysis, have shown great merit for identifying such apps, which mainly rely [...] Read more.
Insecure applications (apps) are increasingly used to steal users’ location information for illegal purposes, which has aroused great concern in recent years. Although the existing methods, i.e., static and dynamic taint analysis, have shown great merit for identifying such apps, which mainly rely on statically analyzing source code or dynamically monitoring the location data flow, identification accuracy is still under research, since the analysis results contain a certain false positive or true negative rate. In order to improve the accuracy and reduce the misjudging rate in the process of vetting suspicious apps, this paper proposes SAMLDroid, a combined method of static code analysis and machine learning for identifying Android apps with location privacy leakage, which can effectively improve the identification rate compared with existing methods. SAMLDroid first uses static analysis to scrutinize source code to investigate apps with location acquiring intentions. Then it exploits a well-trained classifier and integrates an app’s multiple features to dynamically analyze the pattern and deliver the final verdict about the app’s property. Finally, it is proved by conducting experiments, that the accuracy rate of SAMLDroid is up to 98.4%, which is nearly 20% higher than Apparecium. Full article
(This article belongs to the Special Issue Information Theory and Deep Neural Networks)
►▼ Show Figures

Figure 1

19 pages, 1616 KiB  
Article
Compression Helps Deep Learning in Image Classification
by En-Hui Yang, Hossam Amer and Yanbing Jiang
Entropy 2021, 23(7), 881; https://doi.org/10.3390/e23070881 - 10 Jul 2021
Cited by 19 | Viewed by 4894
Abstract
The impact of JPEG compression on deep learning (DL) in image classification is revisited. Given an underlying deep neural network (DNN) pre-trained with pristine ImageNet images, it is demonstrated that, if, for any original image, one can select, among its many JPEG compressed [...] Read more.
The impact of JPEG compression on deep learning (DL) in image classification is revisited. Given an underlying deep neural network (DNN) pre-trained with pristine ImageNet images, it is demonstrated that, if, for any original image, one can select, among its many JPEG compressed versions including its original version, a suitable version as an input to the underlying DNN, then the classification accuracy of the underlying DNN can be improved significantly while the size in bits of the selected input is, on average, reduced dramatically in comparison with the original image. This is in contrast to the conventional understanding that JPEG compression generally degrades the classification accuracy of DL. Specifically, for each original image, consider its 10 JPEG compressed versions with their quality factor (QF) values from {100,90,80,70,60,50,40,30,20,10}. Under the assumption that the ground truth label of the original image is known at the time of selecting an input, but unknown to the underlying DNN, we present a selector called Highest Rank Selector (HRS). It is shown that HRS is optimal in the sense of achieving the highest Top k accuracy on any set of images for any k among all possible selectors. When the underlying DNN is Inception V3 or ResNet-50 V2, HRS improves, on average, the Top 1 classification accuracy and Top 5 classification accuracy on the whole ImageNet validation dataset by 5.6% and 1.9%, respectively, while reducing the input size in bits dramatically—the compression ratio (CR) between the size of the original images and the size of the selected input images by HRS is 8 for the whole ImageNet validation dataset. When the ground truth label of the original image is unknown at the time of selection, we further propose a new convolutional neural network (CNN) topology which is based on the underlying DNN and takes the original image and its 10 JPEG compressed versions as 11 parallel inputs. It is demonstrated that the proposed new CNN topology, even when partially trained, can consistently improve the Top 1 accuracy of Inception V3 and ResNet-50 V2 by approximately 0.4% and the Top 5 accuracy of Inception V3 and ResNet-50 V2 by 0.32% and 0.2%, respectively. Other selectors without the knowledge of the ground truth label of the original image are also presented. They maintain the Top 1 accuracy, the Top 5 accuracy, or the Top 1 and Top 5 accuracy of the underlying DNN, while achieving CRs of 8.8, 3.3, and 3.1, respectively. Full article
(This article belongs to the Special Issue Information Theory and Deep Neural Networks)
►▼ Show Figures

Figure 1

19 pages, 751 KiB  
Article
DLSTM-Based Successive Cancellation Flipping Decoder for Short Polar Codes
by Jianming Cui, Wenxiu Kong, Xiaojun Zhang, Da Chen and Qingtian Zeng
Entropy 2021, 23(7), 863; https://doi.org/10.3390/e23070863 - 6 Jul 2021
Cited by 3 | Viewed by 2513
Abstract
Polar code has been adopted as the control channel coding scheme for the fifth generation (5G), and the performance of short polar codes is receiving intensive attention. The successive cancellation flipping (SC flipping) algorithm suffers a significant performance loss in short block lengths. [...] Read more.
Polar code has been adopted as the control channel coding scheme for the fifth generation (5G), and the performance of short polar codes is receiving intensive attention. The successive cancellation flipping (SC flipping) algorithm suffers a significant performance loss in short block lengths. To address this issue, we propose a double long short-term memory (DLSTM) neural network to locate the first error bit. To enhance the prediction accuracy of the DLSTM network, all frozen bits are clipped in the output layer. Then, Gaussian approximation is applied to measure the channel reliability and rank the flipping set to choose the least reliable position for multi-bit flipping. To be robust under different codewords, padding and masking strategies aid the network architecture to be compatible with multiple block lengths. Numerical results indicate that the error-correction performance of the proposed algorithm is competitive with that of the CA-SCL algorithm. It has better performance than the machine learning-based multi-bit flipping SC (ML-MSCF) decoder and the dynamic SC flipping (DSCF) decoder for short polar codes. Full article
(This article belongs to the Special Issue Information Theory and Deep Neural Networks)
►▼ Show Figures

Figure 1

Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:
 
Displaying articles 1-4
Back to TopTop