entropy-logo

Journal Browser

Journal Browser

Deciphering the Link Between Information and Interpretability in Deep Learning and Artificial Intelligence

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Information Theory, Probability and Statistics".

Deadline for manuscript submissions: 31 May 2025 | Viewed by 1005

Special Issue Editors


E-Mail Website
Guest Editor
U.S. Army Engineer Research and Development Center, 3909 Halls Ferry Road, Vicksburg, MS 39180, USA
Interests: statistical physics; mathematical biology; information theory; network science; dynamical systems

E-Mail Website
Guest Editor
U.S. Army Engineer Research and Development Center, 3909 Halls Ferry Road, Vicksburg, MS 39180, USA
Interests: statistical physics; active soft matter; information theory; dynamical systems

Special Issue Information

Dear Colleagues,

Deep learning models have emerged as a way to reliably identify patterns and correlations in large, complex datasets, but the application of these models to increasingly complex tasks, such as spatiotemporal object tracking or synthetic data generation, has necessitated increasingly sophisticated model architectures whose emergent capabilities are often difficult to understand and interpret from a first principles perspective. Information theory, with its focus on quantifying correlations and uncertainty within datasets, is fertile ground for novel investigations into the ways in which the structure and parameters of a deep learning network synergize with each other to extract predictive patterns within a dataset. In this Special Issue, we are seeking manuscripts that leverage information theory or other statistical or correlative metrics to interrogate how the robust and multifaceted functionality of deep learning architectures or artificial intelligence emerges from the iteration of relatively straightforward mathematical operations that are individually agnostic to the particulars of the training data. As society considers artificial intelligence algorithms a replacement for human productivity, there remains an unacceptable level of mystery involving the mechanisms of their remarkable predictive and even creative capabilities. It is our hope that this Special Issue might help to bridge this gap and bring the state of science more in line with the current state of the art.

Dr. Michael L. Mayo
Dr. Kevin R. Pilkiewicz
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • deep learning
  • neural networks
  • interpretability
  • statistical analysis
  • information theory
  • feature analysis
  • object detection/tracking
  • generative artificial intelligence

Benefits of Publishing in a Special Issue

  • Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
  • Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
  • Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
  • External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
  • e-Book format: Special Issues with more than 10 articles can be published as dedicated e-books, ensuring wide and rapid dissemination.

Further information on MDPI's Special Issue policies can be found here.

Published Papers (1 paper)

Order results
Result details
Select all
Export citation of selected articles as:

Research

16 pages, 2643 KiB  
Article
The Geometry of Concepts: Sparse Autoencoder Feature Structure
by Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun and Max Tegmark
Entropy 2025, 27(4), 344; https://doi.org/10.3390/e27040344 - 27 Mar 2025
Viewed by 835
Abstract
Sparse autoencoders have recently produced dictionaries of high-dimensional vectors corresponding to the universe of concepts represented by large language models. We find that this concept universe has interesting structure at three levels: (1) The “atomic” small-scale structure contains “crystals” whose faces are parallelograms [...] Read more.
Sparse autoencoders have recently produced dictionaries of high-dimensional vectors corresponding to the universe of concepts represented by large language models. We find that this concept universe has interesting structure at three levels: (1) The “atomic” small-scale structure contains “crystals” whose faces are parallelograms or trapezoids, generalizing well-known examples such as (man:woman::king:queen). We find that the quality of such parallelograms and associated function vectors improves greatly when projecting out global distractor directions such as word length, which is efficiently performed with linear discriminant analysis. (2) The “brain” intermediate-scale structure has significant spatial modularity; for example, math and code features form a “lobe” akin to functional lobes seen in neural fMRI images. We quantify the spatial locality of these lobes with multiple metrics and find that clusters of co-occurring features, at coarse enough scale, also cluster together spatially far more than one would expect if feature geometry were random. (3) The “galaxy”-scale large-scale structure of the feature point cloud is not isotropic, but instead has a power law of eigenvalues with steepest slope in middle layers. We also quantify how the clustering entropy depends on the layer. Full article
Show Figures

Figure 1

Back to TopTop