Exploring Neural Network Hidden Layer Activity Using Vector Fields ^†

Abstract

Deep Neural Networks are known for impressive results in a wide range of applications, being responsible for many advances in technology over the past few years. However, debugging and understanding neural networks models’ inner workings is a complex task, as there are several parameters and variables involved in every decision. Multidimensional projection techniques have been successfully adopted to display neural network hidden layer outputs in an explainable manner, but comparing different outputs often means overlapping projections or observing them side-by-side, presenting hurdles for users in properly conveying data flow. In this paper, we introduce a novel approach for comparing projections obtained from multiple stages in a neural network model and visualizing differences in data perception. Changes among projections are transformed into trajectories that, in turn, generate vector fields used to represent the general flow of information. This representation can then be used to create layouts that highlight new information about abstract structures identified by neural networks.

1. Introduction

Given their ability to abstract high-level patterns and model data beyond most heuristics [1], Deep Neural Networks (DNNs) are currently among the state-of-the-art techniques for the analysis of large-scale, complex datasets. Despite the prevalence of DNNs in different domains, such as natural language processing, face and speech recognition, and generation of artificial data, its success heavily depends on the right choice of hyperparameters and architecture.

In recent years, visualization strategies have become increasingly popular in the research community to help analysts interpret the results and support the improvement of DNNs. One of the most popular visualization strategies, multidimensional projection techniques [2], has reportedly attained relative success in helping users explore and explain what happens inside DNNs [3,4,5]. These techniques aim to generate lower-dimensional visual representations capable of preserving multidimensional data structure, such as relationships between data instances or clusters’ presence. However, the currently available techniques are somewhat limited when exploring sequential processes inside neural networks, such as the state of hidden layers during training or the shaping of high-level representations as data flows through different layers of a network. This information can provide insights to important issues, such as determining different layers’ contributions in discriminating certain objects or tracking noteworthy behavior during training, to better understand unusual data.

In this paper, we present a novel approach to visualize the hidden structure of DNNs, designed to aid in understanding abstract high-level representations inside a model and how they are formed. In our approach, we project data extracted from various states of a neural network and estimate a common space between projections by computing a vector field that can show how these projections relate to one another. We show how this space can be used in visualization, as well as possibilities in identifying progress and tracking down meaningful changes in the neural network as training epochs go by or as data flows through the model. This paper is an extension of work presented at the IVAPP 2020 conference [6], refining dimensionality reduction methodology by employing a general approach to projection alignment [7], developing new visual representations for vector field and trajectory data incorporated to sequential projections, and presenting new experiments that highlight these properties.

Our main contributions are:

• A projection-based visual representation suited to represent sequential aspects of DNNs, reducing movement clutter while keeping distances meaningful;
• A visual transition space and vector field between two or more sequential projections that can be used to represent flow or evolution.

The remainder of the paper is organized as follows. In Section 2, we discuss related work regarding visualization techniques for neural networks. In Section 3, we formalize the problem and present a complete overview of our solution. In Section 4, we present an extensive set of experiments to show our solution in practice. In Section 5, we discuss our limitations, and the paper is concluded in Section 6.

2. Related Work

Artificial Neural Networks (ANNs) are structures composed of groups of simple and complex cells. Simple cells are responsible for extracting basic features, while complex ones combine local features producing abstract representations [8]. This structure hierarchically manipulates data through layers (complex cells), each using a set of processing units or neurons (simple cells) to extract local features. In classification tasks, each neuron divides input data space using a linear function (i.e., hyperplane), which is positioned to obtain the best separation as possible between labels of different classes. Thus, the connections among processing units are responsible for combining the half-spaces built up by those linear functions to produce nonlinear separability of data spaces [1]. Deep Neural Networks (DNNs) are artificial neural network models that contain a large number of layers between input and output, generating more complex representations. Such networks are called convolutional neural networks (CNNs) when convolutional filters are employed.

In the past few years, the use of visualization tools and techniques to support the understanding of neural network models has become more prolific, with many different approaches focusing on exploring and explaining different aspects of DNN training, topology, and parametrization [9]. As deep models grow more complex and sophisticated, understanding what happens to data inside these systems is quickly becoming key to improving their efficiency and designing new solutions.

When exploring layers of a DNN, a common source of data are the hidden layer activations: the output value of each neuron of a given layer when subjected to a data instance (input). Many DNN visualization approaches are focused on understanding the high-level abstract representations that are formed in hidden layers. This is often attained by transferring the activations of hidden layer neurons back to the feature space, as defined by the Deconvnet [10] and exemplified by applications such as the Deep Dream [11]. Commonly associated with CNNs, techniques based on this approach often try to explain and represent which feature values in a data object generate activations in certain parts of hidden layers. The Deconvnet is capable of reconstructing input data (images) at each CNN layer to show the features extracted by filters, supporting the detection of incidental problems based on user inspections.

Other techniques focus on identifying content that activates filters and hidden layers. Simonyan et al. [12] developed two visualization tools based on Deconvnet to support image segmentation, allowing feature inspection and summarization of produced features. Zintgraf et al. [13] introduced a feature-based visualization tool to assist in determining the impact of filter size on classification tasks and identifying how the decision process is conducted. Erhan et al. [14] proposed a strategy to identify features detected by filters after their activation functions, allowing the visual inspection of the impact of network initialization as well as if features are humanly understandable. Similarly, Mahendran et al. [15] presented a triple visualization analysis method to inspect images. Babiker et al. [16] also proposed a visual tool to support the identification of unnecessary features filtered in the layers. Liu et al. [17] present a system capable of showing a CNN as an acyclic graph with images describing each filter.

Other methods aim to explore the effects of different parameter configurations in training, such as regularization terms or optimization constraints [5]. These can also be connected to different results in layer activations or classification outcomes. Some techniques are designed to help evaluate the effectiveness of specific network architectures, estimating what kind of abstraction can be learned in each section, such as the approach described by Yosinki et al. [18].

The research previously described is focused on identifying and explaining what representations are generated. However, it is also important to understand how those representations are formed, regarding both the training process and the flow of information inside a network. Comprehending these aspects can lead to improvements in network architecture and the training process itself. The DeepEyes framework, developed by Pezzotti et al. [19], provides an overview of DNNs identifying when a network architecture requires more or fewer filters or layers. It employs scatterplots and heatmaps to show filter activations and allows the visual analysis of the feature space. Kahng et al. [20] introduce a method to explore the features produced by CNN’s projecting activation distances and presenting a neuron activation heatmap for specific data instances. However, these techniques are not designed for projecting multiple transition states, and their projection methods require complex parametrization to show the desired information.

Multidimensional projections (or dimensionality reduction techniques) [2] are popular tools to aid the study of how abstract representations are generated inside ANNs. Specific projection techniques, such as the UMAP [21], were developed particularly with machine learning applications in mind. While dimensionality reduction techniques are generally used in ANN studies to illustrate model efficacy [4,5,22,23,24], Rauber et al. [3] showed their potential on providing valuable visual information on DNNs to improve models and observe the evolution of learned representations. Projections were used to reveal hidden layer activations for test data before and after training, highlighting the effects of training, the formation of clusters, confusion zones, and the neurons themselves, using individual activations as attributes. Despite offering insights on how the network behaves before and after training, the visual representation presented by the authors for the evolution of representations inside the network or the effects of training between epochs displays a great deal of clutter; when analyzing a large number of transition states, information such as the relationships between classes or variations that occur only during intermediate states may become difficult to infer. Additionally, the method used to ensure that all projections share a similar 2D space is prone to problems in alignment. In this paper, we propose a visualization scheme that employs a flow-based approach to offer a representation better suited to show transition stated and evolving data in DNNs. We also briefly address certain pitfalls encountered when visualizing neuron activation data using standard projection techniques, such as t-SNE [25], and discuss why these pitfalls are relevant to our application.

3. Visualizing the Activity Space

Our visual representation is based on gathering hidden layer activation data from a sequence of ANN outputs (steps), then projecting them onto a 2D space while sharing information to ensure semantically similar data remain in similar positions between projections. The movement of the same data instance throughout each projected step generate trajectories, which are then condensed into vector fields that reflect how data flows throughout the sequence. There are two main sequential aspects to be visualized with ANN outputs: how abstract data representations inside the network change and evolve with training and how representations are built and connected as one layer forwards information to the next. For each aspect, a different sequence of hidden layer outputs is needed: outputs from the same given layer during different epochs of training are used to visualize how training changes data representations in that layer, while outputs from different layers from the same network are used to visualize how data representations evolve as layers propagate information.

To build this representation, we first extract activation sets $A\left[1\right],A\left[2\right],\dots ,A\left[T\right]$ representing network outputs from T sequential steps of the process we want to explore. In this paper, we either (a) save a network model at different epochs of training, choose a slicing layer, feed the same set of input data to the saved models, and then save the outputs from the slicing layer, or (b) pick a given network model, slice it at different layers, feed the input data set, and save the outputs from these layers as activation sets.

Once the activation data are extracted, it is projected onto a 2D space using a multidimensional projection technique, obtaining $P\left[1\right],P\left[2\right],\dots ,P\left[T\right]$. To produce coherent projections, an alignment method is applied, ensuring that projections stay similar to each other while preserving original multidimensional distances. Then, positions of the same points in two subsequent projections form movement vectors $\varphi \left[1\right],\dots ,\varphi [T-1]$ that describe how data instances in one output changed to the next. This movement data are joined for all output transitions, generating trajectories (or trails) for each data instance across all T steps, which are then used to compute vector fields. Finally, projections, trajectories, and vector fields are combined to create visualizations. The 2D space shared by all projections is our visual activity space, and trajectories and vector fields describe how network outputs flow through it. Figure 1 summarizes this process.

The following sections detail how data are projected, and how vector fields are generated and visualized in our model.

3.1. Projecting Data

Although we can compare any set of projections to produce vector fields, we need to eliminate changes between projections that do not reflect variations in the high-dimensional data as much as possible. Therefore, the projection technique itself must share information between all data sets to ensure the generation of a synchronized view. Although precise and popular, nonlinear projection techniques generally do not guarantee consistent visual spaces when comparing distinct projections since axes (dimensions) of the original space are not projected into straight lines on visual spaces, and thus cannot be considered equivalent in projected spaces. Some techniques offer a certain degree of control over projection layout, such as fixing control points [26] or selecting the same initialization parameters [25], but often this means a trade-off between local and global distance preservation as they are not designed with this task in mind.

In this context, projections need to be aligned as best as possible, i.e., distances between points in projections must be as similar as possible to those in the original data, while also keeping projections as similar as possible with each other. Currently, there are three different approaches to generating aligned projections of multiple feature sets in literature: Dynamic t-SNE (Dt-SNE) [27], Visual Feature Fusion (VFF) [28], and the generic alignment model [7].

Dt-SNE is a variation of t-SNE capable of minimizing data variability in successive projections to a certain degree to observe changes in multiple DNN states. This is achieved by adding a term to t-SNE’s cost function to approach points representing the same instance in successive projections close to each other. Although attaining better results if compared to the original t-SNE in terms of aligning subsequent projections, it inherits a major problem: the misleading effect of cluster distance and shape in t-SNE projections resulted from the local nature of optimization and how hyperparameters are set up [29]—the observed distance between clusters is unreliable and sensitive to parametrization (perplexity parameter).

The VFF technique was originally developed to fuse different feature representations of data to build a new, user-driven representation. VFF calculates pairwise distances between data points in all feature sets and generates a new pairwise distance table in which the distance between points $x_i,x_j$ is the mean distance $d_T(x_i,x_j)$ across all feature sets T. These distances are used to generate a guide projection p using a force-based model that minimizes metric stress. The technique then proceeds to generate a projection for each feature set, attracting every point towards its counterpart in p at each iteration.

The generic alignment model is the one adopted in this paper. While t-SNE’s global distance preservation is unreliable and VFF’s stress minimization makes it perform poorly for visualizing local neighborhoods, the generic model shows an implementation of UMAP [21], which we view as a reasonable compromise. Additionally, being designed for any gradient descent-based projection technique, this model can be useful for further experiments with other techniques.

3.1.1. Aligning Projections

According to the generic alignment model, the alignment process consists of obtaining samples $F\left[1\right],\dots ,F\left[T\right]$ from activation sets $A\left[1\right]\dots A\left[T\right]$ with the same indexes (i.e., all samples contain data related to the same points) and then calculating projections $P\left[1\right],\dots ,P\left[T\right]\in {\mathbb{R}}^2$ that preserve distance relationships in each set $F\left[k\right]$ as best as possible while aligning themselves to one another. This process implies performing an optimization task considering two restrictions. For each feature set k, the cost function to minimize is given by

$$C\left[k\right]=(1-\lambda )·C_{umap}\left[k\right]+\lambda ·\gamma ·C_{alignment}\left[k\right]$$

(1)

where $C_{umap}\left[k\right]$ is the standard cost function for UMAP (binary cross-entropy), $\lambda$ is the parameter that controls alignment interference ($0.1$ for all experiments in this paper), and $\gamma$ is an internal scaling parameter defined automatically at each iteration. $C_{alignment}\left[k\right]$ is the penalty for inter-projection distance, given by

$$C_{alignment}\left[k\right]=\frac{1}{2N}\sum _{i=1}^{N}d(P_i\left[k\right],\overline{P_i})$$

(2)

where $d(P_{i\left[t\right]},\overline{P_i})$ is the distance between point $P_i$ in the projection k and the current mean position of the same point between all projections ${\overline{P}}_i$.

This optimization is done by gradient descent. To generate aligned projections from hidden layer outputs with widely different number of dimensions, pairwise distances need to be normalized. We performed normalization by dividing attributes by the mean instance norm from every feature set. Once aligned projections are generated, we can proceed to the generation of vector fields.

3.2. Vector Field Generation

The vector fields contained in our visualization are generated using an adapted version of the Vector Field K-means technique, proposed by Ferreira et al. [30]. This technique defines a distance measure $d(X,a)$ between a discrete grid representation of a vector field X and a trajectory a, composed by pairs of time stamp and spatial position $(t,a(t\left)\right)$. Once this distance function is defined, it is minimized to find discrete vector fields that provide the best approximation to a group of trajectories and match each trajectory to a field, similar to the conventional k-means algorithm.

In this approach, a trajectory is represented by a path written as $\alpha :[t_0,t_1]\to {\mathbb{R}}^2$ and a vector field is defined in a domain $\Omega \subset {\mathbb{R}}^2$, i.e., a function $X:\Omega \to {\mathbb{R}}^2$. For all data points, each transition $\varphi$ is considered a path, and time intervals between all steps are presumed equal to $1/T$. Finding the vector field that best matches a set of trajectories $\Phi$ can then be described as an optimization problem of minimizing

$$\begin{array}{c}\hfill E=\underset{X}{min}{\lambda }_L{∥\Delta X∥}^2+\frac{(1-{\lambda }_L)}{N}\sum _{{\alpha }_i\in \Phi }{\int }_{t_0^i}^{t_1^i}{∥X\left({\alpha }_i\left(t\right)\right)-\left({\alpha }^{\prime }\left(t\right)\right)∥}^{2}dt\end{array}$$

(3)

where ${\alpha }^{\prime }\left(t\right)$ is the velocity vector of trajectory $\alpha$ on instant t. The first term is a regularization restriction, where $\Delta$ is the Laplace operator. This restriction ensures smoothness of the resulting vector field, with parameter ${\lambda }_L$ controlling the weight of each term in the equation and therefore determining if the optimization should prioritize smoothness or matching vector field to trajectories.

In the Vector Field k-Means technique, vector fields are generated from trajectory groups and trajectories are then reassigned to the best matching fields at every iteration. Since our focus is to generate only one vector field that approximates a set of trajectories, we employ a simplified version of this technique that fits a single vector field to the trajectory set, with no need for reassignment.

3.3. Vector Field and Trajectory Visualization

The final step in our approach is to combine all information obtained from (1) projections, (2) trajectories, and (3) vector fields components to generate visual representations. For each of these components, we generate different visual representations that are stacked to convey distinct properties of the NN model under analysis. For data projection, we use the conventional scatterplot metaphor with points representing data instances. Since multiple projections $P\left[1\right],P\left[2\right],\dots ,P\left[T\right]$ are available, we allow users to select any projection to compose the visualization, allowing the inspection of the state of given data instances at any moment in time.

Visualization and analysis of vector fields and spatiotemporal data (either abstract or geographic) are research fields with their own bodies of literature, with several proposed methods to solve different tasks [31,32,33,34,35]. In this paper, we adopt two strategies to visualize vector fields: a simple glyph-based approach, showing flow directions, and streamlines, revealing motion across the whole field. With vector field visualization, users can overview data flow inside a NN model through the analysis of instance group formation and separation.

Finally, for trajectories, we employ an approach similar to the one presented by Rauber et al. [3], using bundling to group similar trajectories reducing visual clutter. However, in our approach, we define control points for bundling along the streamlines extracted from the vector field so that bundling follows the flow of data inside the network and allows analysis of the behavior of individual data instances over time. Figure 2 shows an example of these three layers of visualizations used together to represent evolution of representations in the last layer of a NN model during training. In the next section, we detail how this can be used in practice.

4. Results

One typical application of DNNs is data classification, which consists of inferring some model $f:\mathcal{X}\to \mathcal{Y}$ to correctly label unknown data based on a set of known labelled examples $\left(x_i,y_i\right)\in \mathcal{X}\times \mathcal{Y}$ [1]. This learning process is known as supervised learning, in which the model is iteratively adapted according to training examples.

Although it is well established in the literature that DNNs yield outstanding results in classification tasks for different domains, training the network and choosing appropriate parameters can still be a complex, time-consuming process. In this section, we visualize the projected activity space on a few NN models performing classification tasks with the objective of gaining insights on the training process and on how conclusions are drawn when performing classification.

It is important to clarify that these visualization methods are not restricted to a specific architecture or application. These models were chosen as to provide informative examples, and because the classification problem offers an easy way to keep track of instances by coloring points representing them using label data. Experiments were conducted using tensorflow(www.tensorflow.org) python libraries, and visualization systems were built using d3.js(d3js.org).

4.1. MNIST on the MLP model

This example shows projected data obtained from a Multilayer Perceptron (MLP) NN model, consisting of four dense layers with 256 processing units each, using ReLU activation. The output is a 10-unit layer with softmax activation to perform classification on the MNIST dataset [36]. Training is done via stochastic gradient descent (SGD), using learning rate = $0.0001$ and momentum = $0.9$. Output data were gathered from the last hidden layer after $0,1,3$, and 20 training epochs. Accuracy results increased rather quickly, reaching $0.9191,0.9533$, and $0.9800$ on the test set after epochs 1, 3, and 20, respectively. Figure 3 shows projected data and the obtained vector field. The projections show an expansion as the model adapts to classification, and data points move outwards to form groups, in the same way as observed by Rauber et al. [3]. This movement is reflected in the vector field.

An organized flow of instances produces a more intense vector field, while a more disorganized movement implies frequent opposing directions and a muted vector field. This data can be used to display flow between projections: by displaying streamlines, previous movement information can be infused in the visualization without multiple plots or tracking points between projections. Figure 4 shows a projection of the last epoch of training, with streamlines indicating how groups moved to their positions.

While projecting data points over streamlines can offer important insight, additional information can be revealed by directly plotting trajectories. Previous work [3] used trail bundling [37] to reduce visual clutter from multiple trajectories in sequential projections. We defined control points along the generated streamlines that are used to attract nearby trajectories, as can be seen in Figure 5.

The underlying vector field also produces other exploration opportunities: after optimization, there is a resulting error value for each trajectory, which indicates how well it fits to the vector field. This value can be used to explore instances that are not well represented, and identify outliers in the visualization: instances whose movement differs highly to the field surrounding it. Figure 6 shows a highlight of a few trajectories with the highest error, moving against the general flow. Further exploration can be conducted, filtering trajectories and coloring time steps as to understand movement direction and reasoning behind the highlighted behavior.

Figure 7 shows further analysis of a highlighted trajectory. A single instance was selected and its path observed. We can quickly notice what resulted in a high error value from vector field optimization: the instance moved rather quickly and often against the overall flow, crossing large distances and changing groups. Using previous projections as reference, we can identify large instance groups in each area this instance moved to after the first time step. This means the instance was deemed similar to these groups at one point in time, and shifted rather quickly towards a different group afterwards. This behavior is particularly interesting since misclassified examples are often projected on fringe regions of groups, indicating output uncertainty, or projected repeatedly within the same group, being resistant to changes. We can then inspect the data instance to search for clues to this behavior.

4.2. ImageNet on MobileNetV2

This experiment shows our approach applied to a more complex combination of model and data. We used the MobileNetV2 network model [38], pre-trained on Imagenet data [39]. We randomly selected 10 classes (’clasp’, ’delphinium’, ’giant panda’, ’honker’, ’ice cream’, ’lunch’, ’newfoundland’, ’outbuilding’, ’puffin’, ’rose’) from the data set and fine-tuned a new top layer to classify them. Input images were resized to 224 × 224 pixels, resulting in a higher number of attributes when gathering intermediate layer outputs when compared to the previous experiment. MobileNetV2 has a depth of over 150 layers, and while it is possible to generate projections for all of them and visualize them as needed, we opted to select the last merging layer from every two network blocks (’add’ layers), as well as the last global pooling layer, to simplify description. A sample of 1000 data instances from the 10 selected classes was used as input for intermediate layer output projections. All convolutional output arrays were flattened and data were projected using the aligned UMAP model described in Section 3.1.1. Results can be seen in Figure 8.

Our goal with this experiment was to show the forming of abstract representations and knowledge as data flows through the network. While useful information can be gathered by examining projections side by side, this visualization can quickly get out of hand, as larger numbers of projections may need to be generated to understand deeper or more complex models. Figure 9a shows a single projection of the last hidden layer, with streamlines representing previous flow in the background. Certain flow details come to light, such as movement being stronger towards the lower-left side of the image, as the right side contains classes still not fully separated (9 and 2, 5 and 6).

Finally, Figure 9b shows trajectories for this experiment. As the number of projected steps gets higher, the more cluttered the trajectory visualization becomes. This can be solved by dividing trajectories in multiple steps, such as observing how separation in the central section was achieved for the first layers and how expansion was conducted afterwards as two separate images.

5. Limitations

The work described in this paper has certain limitations: one of them is the fact that our projections are forced to be aligned; forcing the layout to keep an aligned state can generate distortions in projected data, especially if the analyzed feature sets offer highly conflicting distance relations or form different structures. Therefore, care must be taken as to not interpret alignment restrictions as patterns in data. The trade-off between alignment and projection fidelity also has a large impact in the generation of vector fields, as an alignment parameter configuration that is too restrictive may result in vector fields that miss too much information, while not being restrictive enough will result in vector fields that show movement that is not relevant. Additionally, outputs from the employed vector field approximation method are not always visually intuitive: for instance, the most intense areas (largest vector norms) in a field will not necessarily be ones with high trajectory activity. Additionally, a single vector field may not be enough to display subtleties of some networks. There is the possibility of generating a layered visualization from multiple vector fields, in order to estimate and explore more complex activity spaces.

6. Conclusions

In this paper, we presented a new approach for projection-based ANN hidden layer visualization that uses trajectories and vector fields to provide insights on how knowledge is generated in a DNN through training and how abstract representations are formed between layers. The presented work offers a flow-based model to represent a transition space between a sequence of projections in order to remove point-based clutter, and incorporate this model into analysis to discover useful information about model and data. Observing trajectories between layers can help understand the importance of each layer in separating certain instances in the data set, and object movement can highlight the effects of a certain training period. Exploring movement throughout the sequence can highlight objects that the model finds difficult to analyze, or objects that are not being perceived in an expected manner.

This approach is not limited to ANN visualization, since any sequence of multidimensional projections can be explored in this manner, such as projecting any pipeline that processes data in multiple steps. The creation of a visual space that ties projections together may provide support to other visualization methods in the future.

We performed experiments that aim to show the usefulness of our method, and how it may lead to knowledge and model improvement. Our analysis was able to go further in certain aspects of the training process of neural networks, attempting to explain subtle aspects of how knowledge is generated in a DNN system.

Future research directions include investigating how projected vector fields relate to other aspects of neural network interpretation methods, such as filter visualization in CNNs, or recurrent models: almost any layer output from ANNs can be projected, and all differences between a sequence of outputs should have an effect on the resulting vector field. Work can also be conducted in finding correlations between movement observed in vector fields and features built by the network, such as identifying neurons (or convolutional filters) that are the most responsible for generating certain movement patterns in the vector field visualization.