An Adaptive Homeostatic Algorithm for the Unsupervised Learning of Visual Features

Abstract

The formation of structure in the visual system, that is, of the connections between cells within neural populations, is by and large an unsupervised learning process. In the primary visual cortex of mammals, for example, one can observe during development the formation of cells selective to localized, oriented features, which results in the development of a representation in area V1 of images’ edges. This can be modeled using a sparse Hebbian learning algorithms which alternate a coding step to encode the information with a learning step to find the proper encoder. A major difficulty of such algorithms is the joint problem of finding a good representation while knowing immature encoders, and to learn good encoders with a nonoptimal representation. To solve this problem, this work introduces a new regulation process between learning and coding which is motivated by the homeostasis processes observed in biology. Such an optimal homeostasis rule is implemented by including an adaptation mechanism based on nonlinear functions that balance the antagonistic processes that occur at the coding and learning time scales. It is compatible with a neuromimetic architecture and allows for a more efficient emergence of localized filters sensitive to orientation. In addition, this homeostasis rule is simplified by implementing a simple heuristic on the probability of activation of neurons. Compared to the optimal homeostasis rule, numerical simulations show that this heuristic allows to implement a faster unsupervised learning algorithm while retaining much of its effectiveness. These results demonstrate the potential application of such a strategy in machine learning and this is illustrated by showing the effect of homeostasis in the emergence of edge-like filters for a convolutional neural network.

1. Introduction: Reconciling Competition and Cooperation

The architecture of the visual system implements a complex dynamic system that operates at different time scales. One of its properties is to succeed in representing information quickly, while optimizing this encoding in the long-term. Respectively, these correspond to the coding and learning time scales. In the case of the mammalian primary visual cortex (V1) for instance, the results of Hubel & Wiesel [1] show that cells of V1 have predominantly relatively localized receptive fields which are selective at different orientations. As such, this rapid coding of the retinal image, of the order of 50 ms in humans, transforms the raw visual information into a rough “sketch” that represents the outlines of objects in the image by using elementary edge-like features. An important aspect of this internal representation is that it is “sparse”: for most natural images, only a relatively small number of features (also called atoms) are necessary to describe the input [2]. Thus, the coding step consists in choosing the right encoder that selects as few features as possible among a collection of them (called the dictionary). Amazingly, Olshausen & Field [3] show that when enforcing a sparse prior on the encoding step, such edge-like filters are obtained using a simple Hebbian unsupervised learning strategy.

Additionally, recent advances in machine learning, and especially on unsupervised learning, have shed new light on the functioning of the underlying biological neural processes. By definition, unsupervised learning aims at learning the best dictionary to represent the input image autonomously, that is, without using other external knowledge, such as in supervised or reinforcement learning. Algorithms that include such a process as the input to classical, supervised deep-learning show great success in tasks like image denoising [4] or classification [5,6]. A variant consists of forcing the generated representation to be sparsely encoded [7], whether by adding a penalty term to the optimized cost function or by encoding each intermediate representation by a pursuit algorithm [8]. Interestingly, [8] proposes a model of Convolutional Sparse Coding (CSC) tightly connected with a Convolutional Neural Network (CNN), so much that the forward pass of the CNN is equivalent to a CSC with a thresholding pursuit algorithm. These unsupervised algorithms are equivalent to a gradient descent optimization over an informational-type coding cost [9]. This cost makes it then possible to quantitatively evaluate the joint exploration of new learning or coding strategies. As such, this remark shows us that unsupervised learning consists of two antagonistic mechanisms, a long time scale that corresponds to the learning and exploration of new components and a faster scale that corresponds to coding, and that both are interdependent.

However, when exploring such algorithms, this convergence may fail to reach a global optimum. In particular, we identified that in simulations for which we aim at comparing the model with the biological substrate, such as when the number of neurons increases, the convergence gradually degenerated (see Figure 1A, “None”). An aspect often ignored in this type of learning is the set of homeostasis processes that control the average activity of neurons within a population. Indeed, there is an intrinsic complexity in unsupervised dictionary learning algorithms. On the one side, neurons are selected by the Sparse Hebbian Learning algorithm by selecting those with maximal activity. This implements a competition within neurons in a population for selecting the one which best matches the visual input. On the other hand, as the learning reinforces the match between the neuron’s response and the visual feature, a regulation process is necessary to avoid the case where only one neuron learns and the other neurons are never selected. Indeed, in such a case, the selection of this neuron would be certain and the surprise associated to this representation would be null. Such homeostatic process thus implements a form of cooperation which aims at optimizing the competition across neurons. But how to adapt the regularization parameter of each atom to make sure no atoms are wasted because of improper regularization settings?

In the original SparseNet algorithm of sparse unsupervised learning [10], homeostasis is implemented as a heuristic that prevents the average energy of each coefficient from diverging. In the majority of present unsupervised learning algorithms, it takes the form of a normalization, that is, an equalization of the energy of each atom in the dictionary [11]. In general, the neural mechanisms of homeostasis are at work in many components of the neural code and are essential to the overall transduction of neural information. For example, the subnetworks of glutamate and GABA-type neurons may regulate the overall activity of neural populations [12]. Such mechanisms could be tuned to balance the contribution of the excitatory populations with respect to that of inhibitory populations. As a consequence, this creates a so-called balanced network, which may explain many facets of the properties of the primary visual cortex [13], such as criticality and scale invariant processing of information in cortical networks, including adaptation. Such a balance may be important to properly represent distributions of activities within a population. This has been demonstrated to be beneficial for image categorization [6]. At the modeling level, these mechanisms are often implemented in the form of normalization rules [14], which are considered as the basis of a normative theory to explain the function of the primary visual cortex [15]. However, when extending such models using unsupervised learning, most effort is focused in showing that the cells’ selectivity has the same characteristics than those observed in neurophysiology [16,17,18]. Other algorithms use nonlinearities that implicitly implement homeostatic rules in neuromimetic algorithms [19] or spiking neurons [20]. These nonlinearities are mainly used in the output of successive layers of deep learning networks that are nowadays widely used for image classification or artificial intelligence. However, most of these nonlinear normalization rules are based on heuristics mimicking neural mechanisms but are not justified as part of the global problem underlying unsupervised learning. Framing this problem in a probabilistic framework allows to consider in addition to coding and learning the intermediate time scale of homeostasis and allows to associate it to an adaptation mechanism [21]. Our main argument is that, compared to classical [10] or Deep Learning approaches, including an homeostatic process optimizes unsupervised learning at both the coding and learning time scales and allows for the implementation of fast algorithms compatible with the performance of biological networks.

In this paper, we will first define a simple algorithm for controlling the selection of coefficients in sparse coding algorithms based on a set of nonlinear functions similar to generic neural gain normalization mechanisms. Such functions will be used to implement a homeostasis mechanism based on histogram equalization by progressively adapting these nonlinear functions. This algorithm will extend an already existing algorithm of unsupervised sparse learning [22] to a more general setting. We will show quantitative results of this optimal algorithm by applying it to different pairs of coding and learning algorithms. Second, we will propose a simplification of this homeostasis algorithm based on the activation probability of each neuron, thanks to the control of the slope of its corresponding Rectifying Linear Unit (ReLU). We show that it yields similar quantitative results as the full homeostasis algorithm and that it converges more rapidly than classical methods [10,23]. We designed our computational architecture to be able to quantitatively cross-validate for every single hyperparameter. All these scripts are available as open-sourced code, including the Supplementary Material. Finally, we will conclude by showing an application of such an adaptive algorithm to CNNs and discuss on its development in real-world architectures.

2. Unsupervised Learning and the Optimal Representation of Images

Visual items composing natural images are often sparse, such that knowing a model for the generation of images, the brain may use this property to represent images using only a few of these items. Images are represented in a matrix $\mathbf{y}={\left({\mathbf{y}}_k\right)}_{k=1}^K\in {\mathbb{R}}^{K\times M}$ as a batch of K vectorial samples (herein, we will use a batch size of $K=256$), where each image is raveled along $M={21}^2=441$ pixels. We use image patches drawn from large images of outdoor scenes, as provided in the “kodakdb” database which is available in the project’s repository. These are circularly masked to avoid artifacts (see Annex (https://spikeai.github.io/HULK/#Loading-a-database)). Each ${\mathbf{y}}_{k,j}\in \mathbb{R}$ is the corresponding luminance value. In the context of the representation of natural images, let us assume the generic Generative Linear Model, such that for any sample k the image was generated as ${\mathbf{y}}_k={\Phi }^T{\mathbf{a}}_k+ϵ$, where by definition, the N coefficients are denoted by ${\mathbf{a}}_k={\left({\mathbf{a}}_{k,i}\right)}_{i=1}^N\in {\mathbb{R}}^N$ and the dictionary by $\Phi \in {\mathbb{R}}^{N\times M}$. Finally, $ϵ\in {\mathbb{R}}^M$ is a Gaussian iid noise, which is normal without loss of generality by scaling the norm of the dictionary’s rows. By understanding this model, unsupervised learning aims at finding the least surprising causes (the parameters ${\widehat{\mathbf{a}}}_k$ and $\Phi$) for the data ${\mathbf{y}}_k$. In particular, the cost may be formalized in probabilistic terms as [10]

$$\begin{array}{cc}\hfill F& \approx {\langle -log\left[p\left({\mathbf{y}}_k\right|{\widehat{\mathbf{a}}}_k,\Phi )p\left({\widehat{\mathbf{a}}}_k\right)\right]\rangle }_{k=1\dots K}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & =\langle \frac{1}{2}\left|\right|{\mathbf{y}}_k-\Phi {\widehat{\mathbf{a}}}_k{\left|\right|}_2^2-logp\left({\widehat{\mathbf{a}}}_k\right){\rangle }_{k=1\dots K}\hfill \end{array}$$

(2)

Such hypothesis allows us to define, in all generality, the different costs that are optimized in most existing models of unsupervised learning. Explicitly, the representation is optimized by minimizing a cost defined on prior assumptions on representation’s sparseness, that is on $logp\left({\widehat{\mathbf{a}}}_k\right)$. For instance, learning is accomplished in SparseNet [10] by defining a sparse prior probability distribution function for each coefficients in the factorial form $logp\left({\mathbf{a}}_k\right)\sim -\beta {\sum }_{i}log(1+\frac{a_i^2}{{\sigma }^2})$, where $\beta$ corresponds to the steepness of the prior and $\sigma$ to its scaling (see Figure 13.2 from the work by the authors of [24]). Then, knowing this sparse solution, learning is defined as slowly changing the dictionary using Hebbian learning. Indeed, to compute the partial derivative of F with respect to $\Phi$, we have simply:

$$\begin{array}{cc}\hfill \frac{\partial }{\partial \Phi }F& ={\langle \frac{1}{2}\frac{\partial }{\partial {\Phi }_i}[{({\mathbf{y}}_k-{\Phi }^T{\widehat{\mathbf{a}}}_k)}^T({\mathbf{y}}_k-{\Phi }^T{\widehat{\mathbf{a}}}_k)]\rangle }_{k=1\dots K}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & ={\langle {\widehat{\mathbf{a}}}_k({\mathbf{y}}_k-{\Phi }^T{\widehat{\mathbf{a}}}_k)\rangle }_{k=1\dots K}.\hfill \end{array}$$

(4)

This allows to define unsupervised learning as the (stochastic) gradient descent using this equation. Similarly to Equation (17) in the work by the authors of [10] or to Equation (2) in the work by the authors of [25], the relation is a linear “Hebbian” rule [26], as it enhances the weight of neurons proportionally to the activity (coefficients) between pre- and postsynaptic neurons. Note that there is no learning for nonactivated coefficients (for which ${\widehat{\mathbf{a}}}_k=0$). Implementing a stochastic gradient descent, we can also use a (classical) scheduling of the learning rate and a proper initialization of the weights (see Annex (https://spikeai.github.io/HULK/#Testing-two-different-dictionary-initalization-strategies)). The only novelty of this formulation compared to other linear Hebbian learning rules, such as those in the work by the authors of [27], is to take advantage of the sparse (nonlinear) representation, hence the name Sparse Hebbian Learning (SHL). In general, the parameterization of the prior in Equation (2) has major impacts on results of the sparse coding, and thus on the emergence of edge-like receptive fields and requires proper tuning. For instance, a L2-norm penalty term (that is, a Gaussian prior on the coefficients) corresponds to Tikhonov regularization [28] and a L1-norm term (that is, an exponential prior for the coefficients) corresponds to the LASSO convex cost which may be optimized by least-angle regression (LARS) [29] or FISTA [30].

2.1. Algorithm: Sparse Coding with a Control Mechanism for the Selection of Atoms

Concerning the choice of a proper prior distribution, the spiking nature of neural information demonstrates that the transition from an inactive to an active state is far more significant at the coding time scale than smooth changes of the firing rate. This is, for instance, perfectly illustrated by the binary nature of the neural code in the auditory cortex of rats [31]. Binary codes also emerge as optimal neural codes for rapid signal transmission [32]. This is also relevant for neuromorphic systems which transmit discrete, asynchronous events such as a network packet or an Address-Event Representation [33]. With a binary event-based code, the cost is only incremented when a new neuron gets active, regardless to its (analog) value. Stating that an active neuron carries a bounded amount of information of $\lambda$ bits, an upper bound for the representation cost of neural activity on the receiver end is proportional to the count of active neurons, that is, to the ${\ell }_0$ pseudo-norm $\left|\right|{\mathbf{a}}_k{\left|\right|}_0=\left|\{i,{\mathbf{a}}_{k,i}\ne 0\}\right|$:

$$\begin{array}{c}\hfill F\approx \langle \frac{1}{2}\left|\right|{\mathbf{y}}_k-\Phi {\mathbf{a}}_k{\left|\right|}_2^2+\lambda \left|\right|{\mathbf{a}}_k{\left|\right|}_0{\rangle }_{k=1\dots K}\end{array}$$

(5)

This cost is similar with information criteria such as the Akaike Information Criteria [34] or distortion rate ([35] p. 571). For $\lambda ={log}_{2}N$, it gives the total information (in bits) to code for the residual (using entropic coding) and the list of spikes’ addresses, as would be sufficient when using a rank-order quantization [36]. In general, the high interconnectivity of neurons (on average of the order of 10,000 synapses per neurons) justifies such an informational perspective with respect to the analog quantization of information in the point-to-point transfer of information between neurons. However, Equation (5) defines a nonconvex cost which is harder to optimize (in comparison to convex formulations in Equation (2) for instance) since the ${\ell }_0$ pseudo-norm sparseness leads to a nonconvex optimization problem, which is “NP-complete” with respect to the dimension M of the dictionary ([35] p. 418).

Still, there are many solutions to this optimization problem and here, we will use a generalized version of the Matching Pursuit (MP) algorithm ([35] p. 422), see Algorithm 1. A crucial aspect of this algorithm is the arg max function as it produces at each step a competition among N neurons (that is, ${log}_{2}N$ bits per event). For this reason, we will introduce a mechanism to tune this competition. For any signal ${\mathbf{y}}_k$ drawn from the database, we get the coefficients ${\mathbf{a}}_k=S({\mathbf{y}}_k;\Psi =\{\Phi ,z,N_0\})$ thanks to the sparse coding step. The parameter $N_0\stackrel{def.}{=}\left|\right|{\mathbf{a}}_k{\left|\right|}_0$ controls the amount of sparsity that we impose to the coding. The novelty of this generalization of MP lies in the scalar functions $z={\left\{z_i\right\}}_{i=1\dots N}$ which control the competition for the best match across atoms. Although the absolute value function is chosen in the original MP algorithm (that is, $\forall i,z_i\left({\mathbf{a}}_k\right)=\left|{\mathbf{a}}_k\right|$), we will define these at a first attempt as the rescaled nonlinear rectified linear unit (ReLU) with gain ${\gamma }_i$: $\forall i,z_i\left({\mathbf{a}}_{k,i}\right)={\gamma }_i\ast {\mathbf{a}}_{k,i}\ast \delta ({\mathbf{a}}_{k,i}>0)$ where $\delta$ is Kronecker’s indicator function. We found, as in the work by the authors of [17], that by using an algorithm like Matching Pursuit (that is using the symmetric function or setting $\forall i,{\gamma }_i=1$ as in [11] for instance), the Sparse Hebbian Learning algorithm could provide results similar to SparseNet, leading to the emergence of Gabor-like edge detectors as is observed in simple cells of the primary visual cortex [37]. One advantage compared to [10] is the nonparametric assumption on the prior based on this more generic ${\ell }_0$ pseudo-norm sparseness. Importantly for our study, we observed that this class of algorithms could lead to solutions corresponding to a local minimum of the full objective function: Some solutions seem as efficient as others for representing the signal but do not represent edge-like features homogeneously (Figure 1A, None). Moreover, using other sparse coding algorithms which are implemented in the sklearn library, we compared the convergence of the learning with different sparse coding algorithms. In particular, we compared the learning as implemented with matching pursuit to that with orthogonal matching pursuit (OMP) [38], LARS or FISTA (see Supplementary Material). For all these sparse coding algorithms, during the early learning step, some cells may learn “faster” than others. These cells have more peaked distributions of their activity and tend to be selected more often (as shown in Figure 1A “None” and quantified in the variability of their distributions in Figure 2A “None”). It is thus necessary to include a homeostasis process that will ensure the convergence of the learning. The goal of this work is to study the specific role of homeostasis in learning sparse representations and to propose a homeostasis mechanism based on the functions $z_i$, which optimizes the learning of an efficient representation.

Algorithm 1 Generalized Matching Pursuit: ${\mathbf{a}}_k=S({\mathbf{y}}_k;\Psi =\{\Phi ,z,N_0\})$

1: set the sparse vector ${\mathbf{a}}_k$ to zero, 2: initialize ${\overline{\mathbf{a}}}_{k,i}=\langle {\mathbf{y}}_k,{\Phi }_i\rangle$ for all i 3: while$\left|\right|{\mathbf{a}}_k{\left|\right|}_0<N_0$do: 4:  select the best match: $i^{\ast }={error}_i\left[z_i\left({\overline{\mathbf{a}}}_{k,i}\right)\right]$ 5:  update the sparse coefficient: ${\mathbf{a}}_{k,i^{\ast }}={\mathbf{a}}_{k,i^{\ast }}+{\overline{\mathbf{a}}}_{k,i^{\ast }}$, 6:  update residual: $\forall i,{\overline{\mathbf{a}}}_{k,i}\leftarrow {\overline{\mathbf{a}}}_{k,i}-{\mathbf{a}}_{k,i^{\ast }}\langle {\Phi }_{i^{\ast }},{\Phi }_i\rangle$.

2.2. Algorithm: Histogram Equalization Homeostasis

Knowing a dictionary and a sparse coding algorithm, we may transform any data sample ${\mathbf{y}}_k$ into a set of sparse coefficients using the above algorithm: ${\mathbf{a}}_k=S({\mathbf{y}}_k;\Psi =\{\Phi ,z,N_0\})$. However, at any step during learning, dictionaries may not have learned homogeneously and may as a result exhibit different distributions for the coefficients. Regrettably, this would not be taken into account in the original cost (see Equation (5)) as we assumed by hypothesis and as in [10] that the components of the sparse vector are identically distributed. To overcome this problem, we may use an additional component to the cost which measures the deviation to this hypothesis:

$$F\approx \langle \frac{1}{2}\left|\right|{\mathbf{y}}_k-\Phi {\mathbf{a}}_k{\left|\right|}_2^2+\lambda \left|\right|{\mathbf{a}}_k{\left|\right|}_0+\mu \mathtt{W}\left({\mathbf{a}}_k\right){\rangle }_{k=1\dots K}$$

(6)

where we define the distance $\mathtt{W}\left({\mathbf{a}}_k\right)$ as the sum of the distances of each individual coefficient’s cumulative probability distribution (that we denote as $P^i$) to the average cumulative probability distribution $P^0=\frac{1}{N}{\sum }_i{P}^i$. Each distance for each atom of index i is defined as the earth mover’s distance (Wasserstein metric with $p=1$), such that $\mathtt{W}\left({\mathbf{a}}_k\right)={\sum }_i{\int }_{a\ge 0}|P^i\left(a\right)-P^0\left(a\right)|da$ [39]. In general, such a distance gives a measure of the solution to the well-known transportation problem between two histograms. In our setting, given a proper value for $\mu$, this gives a lower bound of the estimate of the quantization error. Indeed, as information is coded in the address of neurons (using $\lambda$ bits per coefficient) based on the average distribution of coefficients across neurons, quantization error is lowest when the activity within the neural population is uniformly balanced, that is when each coefficient value is a priori selected with the same probability. When this hypothesis does not hold, we need to transform the value of a coefficient from that which was expected (that is, the average across neurons). It can be shown that this error is proportional to the additional information (in bits) which is necessary to code the vector of coefficients compared to the case where distributions are identically distributed. In particular, a necessary and sufficient condition for minimizing this additional term is that the prior probability of selecting coefficients are identical $\forall (i,j),p\left({\mathbf{a}}_{k,i}\right)=p\left({\mathbf{a}}_{k,j}\right)$. This would result in $\forall i,P^i=P^0$ and thus $\mathtt{W}\left({\mathbf{a}}_k\right)=0$ and cancel the additional term. To reach this optimum, we may use different transformation functions $z_i$ to influence the choice of coefficients such that we may use these functions to optimize the objective cost defined by Equation (6).

To achieve this uniformity, we may define a homeostatic gain control mechanism based on histogram equalization, that is, by transforming coefficients in terms of quantiles by setting $\forall i,z_i\left(a\right)=P^i\left(a\right)\stackrel{def.}{=}Pr(a>a_i)$. Such a transform is similar to the inverse transform sampling which is used to optimize representation in auto-encoders [40] and can be considered as a nonparametric extension of the “reparameterization trick” used in variational auto-encoders [9]. Moreover, it has been found that such an adaptation mechanism is observed in the response of the retina to various contrast distributions [41]. However, an important point to note is that this joint optimization problem between coding and homeostasis is circular as we can not access the true posterior $Pr\left(\mathbf{a}\right)$: Indeed, the coefficients depend on nonlinear coefficients through ${\mathbf{a}}_k=S({\mathbf{y}}_k;\Psi =\{\Phi ,z_i,N_0\})$, whereas the nonlinear functions depend on the (cumulative) distribution of the coefficients. We will make the assumption that such a problem can be solved iteratively by slowly learning the nonlinear functions. Starting with an initial set of nonlinear functions as in None, we will derive an approximation for the sparse coefficients. Then, the function $z_i$ for each coefficient of the sparse vector is calculated using an iterative moving average scheme (parameterized by time constant $1/{\eta }_h$) to smooth its evolution during learning. At the coding level, this nonlinear function is incorporated in the matching step of the matching pursuit algorithm (see Algorithm 1), to modulate the choice of the most probable as that corresponding to the maximal quantile: $i^{\ast }={\arg \max }_i{z}_i\left(a_i\right)$. We will coin this variant as Histogram Equalization Homeostasis (HEH). The rest of this Sparse Hebbian Learning algorithm is left unchanged. As we adapt the dictionaries progressively during Sparse Hebbian Learning, we may incorporate this HEH homeostasis during learning by choosing an appropriate learning rate ${\eta }_h$. To recapitulate the different choices we made from the learning to the coding and the homeostasis, the unsupervised learning can be summarized using the following steps.

We compared qualitatively the set $\Phi$ of receptive filters generated with different homeostasis algorithms (see Figure 1A). A more quantitative study of the coding is shown by comparing the decrease of the cost as a function of the iteration step (see Figure 1B). This demonstrate that forcing the learning activity to be uniformly spread among all receptive fields results in a faster convergence of the representation error as represented by the decrease of the cost F.

2.3. Results: A More Efficient Unsupervised Learning Using Homeostasis

We have shown above that we can find an exact solution to the problem of homeostasis during Sparse Hebbian Learning. However, this solution has several drawbacks. First, it is computationally-intensive on a conventional computer as it necessitates to store each $z_i$ function to store the cumulative distribution of each coefficient. More importantly, it seems that biological neurons seem to rather use a simple gain control mechanism. This can be implemented by modifying the gain ${\gamma }_i$ of the slope of the ReLU function to operate a gradient descent on the cost based on the distribution of each coefficients. Such strategy can be included in the SHL algorithm by replacing line 9 in the learning algorithm (see Algorithm 2) by $z_i\left(a\right)={\gamma }_i·a·\delta (·>0)$. For instance, the strategy in SparseNet [10] assumes a cost on the difference between the observed variance of coefficients $V_i$ as computed over a set of samples compared to a desired value ${\sigma }_g^2$ (and assuming a multiplicative noise parameterized by $\alpha$) :

$$\begin{array}{c}{V}_i\leftarrow (1-{\eta }_h)·V_i+{\eta }_h·1/K\sum _{k=1\cdots K}{a}_{i,k}^2\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & \mathrm{and}{\gamma }_i\leftarrow {\gamma }_i·{\left(\frac{V_i}{{\sigma }_g^2}\right)}^{\alpha }\hfill \end{array}$$

(8)

Algorithm 2 Homeostatic Unsupervised Learning of Kernels: $\Phi =H(\mathbf{y};\eta ,{\eta }_h,N_0)$

1: Initialize the point nonlinear gain functions $z_i$ to similar cumulative distribution functions, 2: Initialize N atoms ${\Phi }_i$ to random points on the M-unit sphere, 3: forT epochs do: 4:  draw a new batch $\mathbf{y}$ from the database of natural images, 5:  for each data point ${\mathbf{y}}_k$ do: 6:   compute the sparse representation vector using sparse coding ${\mathbf{a}}_k=S({\mathbf{y}}_k;\Psi =\{\Phi ,z,N_0\})$, 7:   modify atoms: $\forall i,{\Phi }_i\leftarrow {\Phi }_i+\eta ·{\mathbf{a}}_{k,i}·({\mathbf{y}}_k-\Phi {\mathbf{a}}_k)$, 8:   normalize atoms: $\forall i,{\Phi }_i\leftarrow {\Phi }_i/\left|\right|{\Phi }_i\left|\right|$, 9:    update homeostasis functions: $\forall i,z_i\left(·\right)\leftarrow (1-{\eta }_h)·z_i\left(·\right)+{\eta }_h·\delta ({\mathbf{a}}_{k,i}\le ·)$.

This is similar to the mechanisms of gain normalization proposed by the authors of [14], which were recently shown to provide efficient coding mechanisms by the authors of [42]. However, compared to these methods which manipulate the gain of dictionaries based on the energy of coefficients, we propose to rather use a methodology based on the probability of activation. Indeed, the main distortion that occurs during learning is on higher statistical moments rather than variance, for instance when an atom is winning more frequently during the earliest iterations, its pdf will typically be more kurtotic than a filter that has learned less.

Recently, such an approach was proposed by the authors of [23]. Based on the same observations, the authors proposed to optimize the coding during learning by modulating the gain of each dictionary element based on the recent activation history. They base their Equalitarian Matching Pursuit (EMP) algorithm on a heuristics, which cancels the activation of any filter that was more often activated than a given threshold probability (parameterized by $1+{\alpha }_h$). In our setting, we may compute a similar algorithm using an evaluation of the probability of activation followed by binary gates

$$\begin{array}{c}{p}_i\leftarrow (1-{\eta }_h)·p_i+{\eta }_h·1/K\sum _{k=1\cdots K}\delta (a_{i,k}>0)\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & \mathrm{and}{\gamma }_i=\delta (p_i<N_0/N·(1+{\alpha }_h))\hfill \end{array}$$

(10)

As such, $p_i$ is an approximation of the average activation probability based on a moving average controlled by the learning parameter ${\eta }_h$. Interestingly, they reported that such a simple heuristic could improve the learning, deriving a similar result as we have shown in Figure 1 and Figure 2. Moreover they have shown that such a homeostatic mechanism is more important than optimizing the coding algorithm, for instance by using OMP instead of MP. Again, such strategy can be included in line 9 of the learning algorithm.

Similarly, we may derive an approximate homeostasis algorithm based on the current activation probability, but using an optimization approach on the gain modulation. Ideally, this corresponds to finding ${\gamma }_i$ such that we minimize the entropy $-{\sum }_{i=1\cdots N}{p}_{i}log{p}_i$. However, the sparse coding function $S({\mathbf{y}}_k;\Psi =\{\Phi ,z,N_0\})$, which would allow to compute $p_i$ is not differentiable. A simpler approach is to compute the change of modulation gain that would be necessary to achieve an uniform probability. Indeed, such “equiprobability” is the known solution of the maximum entropy problem, that is when $\forall i,p_i=p_0\stackrel{def.}{=}{N}_0/N$:

$$\begin{array}{c}{p}_i\leftarrow (1-{\eta }_h)·p_i+{\eta }_h·1/K\sum _{k=1\cdots K}\delta (a_{i,k}>0)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \mathrm{and}{\gamma }_i=\frac{log(1/p_i)}{log(1/p_0)}=\frac{log\left(p_i\right)}{log\left(p_0\right)}\hfill \end{array}$$

(12)

where ${\eta }_h$ controls as above the speed of the sliding average for estimating the activation probability. Note that the gain is equal to one if the activation probability reaches the target probability. It becomes excitatory or inhibitory for cells whose probability is, respectively, below or above the target. Assuming an exponential probability distribution function for the sparse coefficients before the thresholding operation, this expression follows as the solution to scale coefficients such that overall each neuron fires with equal probability. We will coin this variant of the algorithm Homeostasis on Activation Probability (HAP). Following these derivations, we quantitatively compared OLS, EMP, and HAP to HEH (see Figure 3). This shows that although EMP slightly outperforms OLS (which itself is more efficient than None, see Figure 2B), HAP proves to be closer to the optimal solution given by HEH. Moreover, we replicated in HAP the result of [23] that while homeostasis was essential in improving unsupervised learning, the coding algorithm (MP vs. OMP) mattered relatively little (see Annex (https://spikeai.github.io/HULK/#Testing-different-algorithms)). Also, we verified the dependence of this efficiency with respect to different hyperparameters (as we did in Figure 2B). Overall, these quantitative results show that the HEH algorithm could be replaced by a simpler and more rapid heuristic, HAP, which is based on activation probability. This would generate a similar efficiency for the coding of patches from natural images.

3. Discussion and Conclusions

One core advantage of sparse representations is the efficient coding of complex multidimensional signals such as images using compact codes. Inputs are thus represented as a combination of few elements drawn from a large dictionary of atoms. A common design for unsupervised learning rules relies on a gradient descent over a cost measuring representation quality with respect to sparseness. This constraint introduces a competition between atoms. In the context of the efficient processing of natural images, we proposed here that such strategies can be optimized by including a proper homeostatic regulation enforcing a fair competition between the elements of the dictionary. We implemented this rule by introducing a nonlinear gain normalization similar to what is observed in biological neural networks. We validated this theoretical insight by challenging this adaptive unsupervised learning algorithm with different heuristics for the homeostasis. Simulations show that at convergence, although the coding accuracy did not vary much, including homeostasis changed, qualitatively, the learned features. In particular, including homeostasis resulted in a more homogeneous set of orientation selective filters, which is closer to what is observed in the visual cortex of mammals [16,17,18]. To further validate these results, we quantitatively compared the efficiency of the different variants of the algorithms, both at the level of homeostasis (homeostatic learning rate, parameters of the heuristics), but also to the coding (by changing M, N or $N_0$) and to the learning (by changing the learning rate, the scheduling or M). This demonstrated that overall, this neuro-inspired homeostatic algorithm provided with the best compromise between efficiency and computational cost.

In summary, these results demonstrate that principles observed in biological neural computations can help improve real-life machine learning algorithms, in particular, for vision. Indeed, by developing this fast learning algorithm, we hope for its use in real-life machine learning algorithms. This type of architecture is economical, efficient and fast. The HAP algorithms uses only ReLUs such that it is easy to be transferred to most deep learning algorithms. Additionally, we hope that this new type of rapid unsupervised learning algorithm can provide a normative theory for the coding of information in low-level sensory processing, whether it is visual or auditory. Moreover, by its nature, this algorithm can easily be extended to convolutional networks such as those used in deep learning neural networks. This extension is possible by extending the filter dictionary by imposing the hypothesis of the invariance of synaptic patterns to spatial translations. Our results on different databases show the stable and rapid emergence of characteristic filters on these different bases (see Figure 4 and Annex (https://spikeai.github.io/HULK/#Testing-different-algorithms)). This result shows a probable prospect of extending this representation and for which we hope to obtain classification results superior to the algorithms existing in the state-of-the-art. As such, empirical evaluations of the proposed algorithms should be extended. For instance, it would be very useful to test for image classification results on standard benchmark datasets.