SemanticCAP: Chromatin Accessibility Prediction Enhanced by Features Learning from a Language Model

Abstract

A large number of inorganic and organic compounds are able to bind DNA and form complexes, among which drug-related molecules are important. Chromatin accessibility changes not only directly affect drug–DNA interactions, but they can promote or inhibit the expression of the critical genes associated with drug resistance by affecting the DNA binding capacity of TFs and transcriptional regulators. However, the biological experimental techniques for measuring it are expensive and time-consuming. In recent years, several kinds of computational methods have been proposed to identify accessible regions of the genome. Existing computational models mostly ignore the contextual information provided by the bases in gene sequences. To address these issues, we proposed a new solution called SemanticCAP. It introduces a gene language model that models the context of gene sequences and is thus able to provide an effective representation of a certain site in a gene sequence. Basically, we merged the features provided by the gene language model into our chromatin accessibility model. During the process, we designed methods called SFA and SFC to make feature fusion smoother. Compared to DeepSEA, gkm-SVM, and k-mer using public benchmarks, our model proved to have better performance, showing a 1.25% maximum improvement in auROC and a 2.41% maximum improvement in auPRC.

1. Introduction

In human cells, genetic and regulatory information is stored in chromatin, which is deoxyribonucleic acid (DNA) wrapped around histones. The chromatin structure has a lot to do with gene transcription, protein synthesis, biochemical processes, and other complex biological expressions. Among them, the binding of small organic and inorganic molecules to the DNA can influence numerous biological processes in which DNA participate. In particular, many anticancer, antibiotic, and antiviral drugs exert their primary biological effects by reversibly interacting with nucleic acids. Therefore, the study of its structure can help us design drugs to control gene expression and to cure diseases [1]. Some regions of the chromatin are open to transcription factors (TFs), RNA polymers (RNAPs), drug molecules, and other cellular materials, while others are tightly entangled together and do not play a role in most cellular processes. These two regions of the chromatin are called open regions and closed regions, which are also known as accessible and inaccessible regions [2]. Measuring the accessibility of chromatin regions can generate clues to gene function that can help us to identify appropriate targets for therapeutic intervention. Meanwhile, monitoring changes in chromatin accessibility can help us to track and understand drug effects. A study [3] found that chromatin accessibility changes at intergenic regions are associated with ovarian cancer drug resistance. Another example is the way that the chromatin opening (increased accessibility) of the targeted DNA satellites can explain how the DNA-binding pyrrole–imidazole compounds that target different Drosophila melanogaster satellites lead to gain- or loss-of-function phenotypes [4]. In recent years, many high-throughput sequencing technologies have been used for the detection of open regions, such as DNase-seq [5], FAIRE-seq [6], and ATAC-seq [7]. However, biological experimental methods are costly and time-consuming and thus cannot be applied to large-scale chemical examinations. These restrictions have promoted the development of calculation methods.

Alongside the progress in computer science, several kinds of sequence-based calculation methods have been proposed to identify functional regions. Simply put, we can divide them into traditional machine learning methods [8,9,10,11,12] and neural network methods [13,14,15,16,17]. Machine learning methods are mainly based on support vector machines (SVM), which perform supervised learning for the classification or regression of data groups. An SVM method [8] based on k-mer features, which are defined as a full set of segments of varying lengths (3–10 bp) in a long sequence, was designed in 2011. This method recognizes enhancers in mammalian cells. Subsequently, the gkm-SVM (gapped k-mer SVM) proposed in 2014 [9] exploited a feature set called interval k-mer features to improve the accuracy and stability of recognition. This method only uses the part of the segments that vary in length, instead of all of the segments. In recent years, with the rapid development of neural networks and the emergence of various deep learning models, a growing number of deep network models have come to be used to solve such problems, where convolutional neural networks (CNNs) [18] and recurrent neural networks (RNNs) [19] are dominant in this regard. A neural network is a computational learning system that uses a network of functions to understand and translate the data input of one form into a desired output, and deep learning is a type of artificial neural networks in which multiple layers of processing are used to extract progressively higher-level features from data. CNNs use the principle of convolution to encode the local information of the data, while RNNs model the sequence with reference to the memory function of the neurons. CNNs are used in DeepBind [13] and DeepSEA [14] to model the sequence specificity of protein binding, and they have both demonstrated significant performance improvements compared to traditional SVM-based methods. Min et al. utilized long short-term memory (LSTM) [15] to predict chromatin accessibility and achieved state-of-the-art results for the time, thus proving the effectiveness of RNNs for DNA sequence problems.

However, we point out that the previous methods have the following shortcomings. First, most of the previous methods are based on k-mer, that is, a segment of length $k$. Specifically, it takes a segment of length $k$ at intervals. The artificial division of the original sequence may destroy the internal semantic information, causing difficulties when learning subsequent models. Second, with the progress being made in language models, we have the ability to learn the interior semantic information of sequences through pre-training. There has been related work on existing methods, such as using GloVe [20] to train the k-mer word vectors. However, these pre-training models are mostly traditional word vector methods. On the one hand, they can only learn the characteristics of the word itself and have no knowledge of the context of DNA sequences [21]. On the other hand, they are limited to a specific dataset and thus cannot be widely applied to other scenarios. Third, traditional CNNs and RNNs have been proven to be unsuitable for long-sequence problems [22]. CNNs, restricted by the size of convolution kernels, fail to learn global information effectively, while RNNs tend to cause gradient disappearance and result in slow training due to the lack of parallelizability when receiving a long input. In contrast, the attention mechanism (Attention) [23] can effectively learn the long-range dependence of sequences and has been widely used in the field of natural language processing.

In response to the above disadvantages, we constructed a chromatin accessibility prediction model called SemanticCAP, which is based on features learning from a language model. The data and code for our system are available at github.com/ykzhang0126/semanticCAP (accessed on 16 February 2022). The SemanticCAP model, trained on DNase-seq datasets, has an ability to predict the accessibility of DNA sequences from different cell lines and thus can be used as an effective alternative to biological sequencing methods such as DNase-seq. At a minimum, our model makes the following three improvements:

• A DNA language model is utilized to learn the deep semantics of DNA sequences and introduces the semantic features in the chromatin accessibility prediction process; therefore, we are able to obtain additional complex environmental information.
• Both the DNA language model and the chromatin accessibility model use character-based inputs instead of k-mer which stands for segments of length $k$. The strategy prevents the information of original sequences from being destroyed.
• The attention mechanism is widely used in our models in place of CNNs and RNNs, making the model more powerful and stable in handling long sequences.

Before formally introducing our method, we will first present some preliminary knowledge, including some common-sense information, theorems, and corollaries.

2. Theories

Theorem 1.

For two standardized distributions using layer normalization (LN), which are denoted as $X_1$and  $X_2$, the concat of them, that is, $X\equiv \left[X_1,X_2\right]$, is still a standardized distribution.

Proof. 

Suppose that $X_1$ has $n$ elements and $X_2$ has $m$ elements. As we all know, LN [24] transforms the distribution $X$ as

$$X\underset{}{\stackrel{\mathrm{LN}}{\to }}\frac{X-\mu }{\sigma }$$

(1)

where $\mu$ and $\sigma$ are the expectation and standard deviation of $X$ respectively. Obviously, for the normalized distribution $X_1$ and $X_2$, we have

$$E\left(X_1\right)=E\left(X_2\right)=0$$

(2)

$$D\left(X_1\right)=D\left(X_2\right)=1$$

(3)

where $E$ stands for the expectation function and $D$ stands for the deviation function. The new distribution $X$ is derived by concating $X_1$ and $X_2$, and thus has $n+m$ elements. Inferring from Equation (2), we have

$$E\left(X\right)=\frac{nE\left(X_1\right)+mE\left(X_2\right)}{n+m}=0$$

(4)

$$\begin{array}{ll}D\left(X\right)& =E\left(X^2\right)-E^2\left(X\right)=E\left(X^2\right)\\ & =\frac{nE\left(X_1^2\right)+mE\left(X_2^2\right)}{n+m}\end{array}$$

(5)

For $X_1$ and $X_2$, we also know that

$$E\left(X_1^2\right)=D\left(X_1\right)+E^2\left(X_1\right)$$

(6)

$$E\left(X_2^2\right)=D\left(X_2\right)+E^2\left(X_2\right)$$

(7)

Substituting Equations (2) and (3) into Equations (6) and (7), and finally into Equation (5), we have

$$D\left(X\right)=1$$

(8)

Equations (4) and (8) demonstrate the standardization of $X$. □

Theorem 2.

For any two distributions $X_1$, $X_2$, there two coefficients ${\lambda }_1$, ${\lambda }_2$that always exist, so that the concat of them, after being multiplied by the two coefficients, respectively, that is$X\equiv \left[{\lambda }_1{X}_1,{\lambda }_2{X}_2\right]$, is a standardized distribution.

Proof. 

Suppose that $X_1$ has $n$ elements and $X_2$ has $m$ elements. We denote the expectation of the two distributions as ${\mu }_1$, ${\mu }_2$, and the variance as ${\sigma }_1^2$, ${\sigma }_2^2$. Notice that ${\lambda }_1$ and ${\lambda }_2$ are all scalars. Now, pay attention to $X$. To prove this theorem, we want $X$ to be a standardized distribution, which requires the expectation of $X$ to be $0$ and the variance to be $1$. Therefore, we can list the following equation set:

$$\left\{\begin{array}{ll}& E\left(X\right)=\frac{n{\lambda }_{1}E\left(X_1\right)+m{\lambda }_{2}E\left(X_2\right)}{n+m}=0\\ & E\left(X^2\right)=\frac{nE\left({\left({\lambda }_1{X}_1\right)}^2\right)+mE\left({\left({\lambda }_2{X}_2\right)}^2\right)}{n+m}\\ & D\left(X\right)=E\left(X^2\right)-E^2\left(X\right)=1\end{array}\right.$$

(9)

At the same time, we have equations similar to Equations (6) and (7), those being:

$$E\left({\left({\lambda }_1{X}_1\right)}^2\right)=D\left({\lambda }_1{X}_1\right)+E^2\left({\lambda }_1{X}_1\right)$$

(10)

$$E\left({\left({\lambda }_2{X}_2\right)}^2\right)=D\left({\lambda }_2{X}_2\right)+E^2\left({\lambda }_2{X}_2\right)$$

(11)

which are easy to calculate according to the nature of expectation and variance. Notice that Equation (9) has two variables and two independent equations, meaning it should be solvable. By calculating Equation (9), we can determine the numeric solution of ${\lambda }_1$ and ${\lambda }_2$ as follows:

$$\left\{\begin{array}{cc}& {\lambda }_1^2=\frac{m\left(n+m\right){\mu }_2^2}{nm{\mu }_2^2\left({\mu }_1^2+{\sigma }_1^2\right)+n^2{\mu }_1^2\left({\mu }_2^2+{\sigma }_2^2\right)}\\ & {\lambda }_2^2=\frac{n\left(n+m\right){\mu }_1^2}{nm{\mu }_1^2\left({\mu }_2^2+{\sigma }_2^2\right)+m^2{\mu }_2^2\left({\mu }_1^2+{\sigma }_1^2\right)}\end{array}\right.$$

(12)

The existence of Equation (12) ends our proof. Actually, we are able to obtain two sets of solutions here because ${\lambda }_1$ can either be positive or negative, and so can ${\lambda }_2$. The signs of ${\lambda }_1$ and ${\lambda }_2$ depend on the signs of ${\mu }_1$ and ${\mu }_2$, which can be easily inferred from the first equation in Equation (9). □

Corollary 1.

For any distributions  $X_1$, $X_2$, $\dots$, $X_n$, their  $n$ coefficients  ${\lambda }_1$, ${\lambda }_2$, $\dots$, ${\lambda }_n$, always exist, so that the concat of them, after being multiplied by the  $n$ coefficients, respectively, that is  $X\equiv \left[{\lambda }_1{X}_1,{\lambda }_2{X}_2,\dots ,{\lambda }_n{X}_n\right]$, is a standardized distribution.

Proof. 

The overall method of proof is similar to that used in Theorem 2. Note that, in this case, we have $n$ variables but only two independent equations, resulting in infinite solutions according to Equation (9). To be more precise, the degree of freedom of our solutions is $n-2$. □

Theorem 3.

In neural networks, for any two tensors  $X$, $Y$ that satisfy  $E\left(X\right)=E\left(Y\right)=0$, the probability of feature disappearance of $X$ after concating and normalizing them is  $\Omega \left(\frac{SD\left(Y\right)}{SD\left(X\right)}\right)$, where  $SD$ represents the standard deviation.

Proof. 

Feature disappearance is defined as a situation where the features are too small. Concretely, for a tensor $X$ and a threshold $t_{FD}$, if the result of a subsequent operation of $X$ is smaller than $t_{FD}$, then the feature disappearance of $X$ occurs. Here, $t_{FD}$ can be an arbitrarily small value, such as ${10}^{-5}$.

Suppose that $X$ has $n$ elements and $Y$ has $m$ elements. We denote the expectation of the two distributions as ${\mu }_1$, ${\mu }_2$, and the variance as ${\sigma }_1^2$, ${\sigma }_2^2$. As stated in the precondition, we already know that

$${\mu }_1={\mu }_2=0$$

(13)

Let $Z\equiv \left[X,Y\right]$ and $Z^{\prime }\equiv \mathrm{LN}\left(Z\right)\equiv \left[X^{\prime },Y^{\prime }\right]$. With the help of Equation (9), we have

$$E\left(Z\right)=0$$

(14)

$$D\left(Z\right)=E\left(Z^2\right)=\frac{n{\sigma }_1^2+m{\sigma }_2^2}{n+m}$$

(15)

We denote $E\left(Z\right)$ as $\mu$ and $D\left(Z\right)$ as ${\sigma }^2$. According to Equation (1), for $X^{\prime }$, we know that

$$E\left(X^{\prime }\right)=E\left(\frac{X-E\left(Z\right)}{\sqrt{D\left(Z\right)}}\right)=\frac{{\mu }_1-\mu }{\sigma }$$

(16)

$$D\left(X^{\prime }\right)=D\left(\frac{X-E\left(Z\right)}{\sqrt{D\left(Z\right)}}\right)=\frac{{\sigma }_1^2}{{\sigma }^2}$$

(17)

We denote $E\left(X^{\prime }\right)$ as ${{\mu }_1}^{\prime }$ and $D\left(X^{\prime }\right)$ as ${{\sigma }_1}^{\prime 2}$. Now, we consider the results of a subsequent operation of $X$, which is ${{\displaystyle \sum }}_{i=1}^n{\lambda }_i{X_i}^{\prime }$. This is very common in convolution, linear, or attention layers. For the result, an observation is

$$\begin{array}{c}{\displaystyle \sum _{i=1}^n}{\lambda }_i{X_i}^{\prime }\le {\displaystyle \sum _{i=1}^n}\left|{\lambda }_i\right|\left|{X_i}^{\prime }\right|\le {\lambda }_m{\displaystyle \sum _{i=1}^n}\left|{X_i}^{\prime }\right|\end{array}$$

(18)

where ${\lambda }_m=\underset{1\le i\le n}{max}\left|{\lambda }_i\right|$. For the convenience of analysis, all $\lambda$ are set to 1. This will not result in a loss of generality because the value scaling from ${\lambda }_m$ to 1 has no effect on the subsequent derivation. Here, we denote $\sum X^{\prime }$ as $S_{X^{\prime }}$. According to the central limit theorem (Lindeberg–Lévy form) [25], we find that $S_{X^{\prime }}$ obeys a normal distribution, that is

$$S_{X^{\prime }}\sim \mathcal{N}\left(n{{\mu }_1}^{\prime },n{{\sigma }_1}^{\prime 2}\right)$$

(19)

For a feature disappearance threshold $t_{FD}$, we want to figure out the probability of $\left|S_{X^{\prime }}\right|<t_{FD}$. Denote this event as $FD$, and we can obtain

$$\begin{array}{ll}Pr\left(FD\right)& =\mathrm{Pr}\left\{\left|S_{X^{\prime }}\right|<t_{FD}\right\}\\ & =\mathrm{Pr}\left\{\left|\frac{S_{X^{\prime }}-n{{\mu }_1}^{\prime }}{\sqrt{n}{{\sigma }_1}^{\prime }}\right|<\frac{t_{FD}-n{{\mu }_1}^{\prime }}{\sqrt{n}{{\sigma }_1}^{\prime }}\right\}\\ & =2\Phi \left(\frac{t_{FD}-n{{\mu }_1}^{\prime }}{\sqrt{n}{{\sigma }_1}^{\prime }}\right)-1\end{array}$$

(20)

where $\Phi$ is the cumulative distribution function (cdf) of the standard normal distribution. Since it is an integral that does not have a closed form solution, we cannot directly analyze it. According to Equations (13), (14) and (16), we know that ${{\mu }_1}^{\prime }=0$. At the same time, we know that $t_{FD}$ is a small number, leading to $\frac{t_{FD}}{\sqrt{n}{{\sigma }_1}^{\prime }}\to 0$. Therefore, we have the equation as follows:

$$\begin{array}{ll}\Phi \left(x\right)& =\Phi \left(0\right)+\phi \left(0\right)x+R_1\left(x\right)\\ & =0.5+\frac{1}{\sqrt{2\pi }}x+o\left(x\right)\end{array}$$

(21)

The formula is a Taylor expansion where $\phi$ is the probability density function (pdf) of the standard normal distribution, $R_1\left(x\right)$ is the Lagrange remainder, and $o\left(x\right)$ is the Peano remainder, standing for a high-order infinitesimal of $x$. Combining Equations (15), (17), (20) and (21), we achieve

$$\begin{array}{ll}Pr\left(FD\right)& =t_{FD}\sqrt{\frac{2\left(k_s{k}_d^2+1\right)}{\pi n\left(k_s+1\right)}}+o\left(k_d\right)\\ & =\Omega \left(K_d\right)\end{array}$$

(22)

where $k_s=\frac{m}{n}$ and $k_d=\frac{{\sigma }_2}{{\sigma }_1}$. The above equation can also be written as $Pr\left(FD\right)=\Omega \left(\frac{SD\left(Y\right)}{SD\left(X\right)}\right)$. □

Corollary 2.

In neural networks, feature disappearance can lead to gradient disappearance.

Proof. 

According to Theorem 3, feature disappearance happens if there exists a tensor $T$ such that $\left|T\right|<t_{FD}$. Similar to the definition of feature disappearance, gradient disappearance is defined as a situation where the gradients are too small. Concretely, for a parameter $C$ with a gradient of $gra{d}_C$ and a threshold $t_{GD}$, if $gra{d}_C$ is smaller than $t_{GD}$, the gradient disappearance of $C$ happens. Here, $t_{GD}$ can be an arbitrarily small value.

Consider a subsequent operation of $T$, which is $T^{\prime }=C{T}^n$, where $n$ stands for the number of layers involved in the calculation. The gradient disappearance happens if

$$\left|gra{d}_C\right|=\left|\frac{\mathrm{d}{T}^{\prime }}{\mathrm{d}C}\right|=\left|T^n\right|={\left|T\right|}^n<t_{GD}$$

(23)

At the same time, we already have ${\left|T\right|}^n<t_{FD}^n$, which means that we simply need to meet the requirements for

$$t_{FD}^n<t_{GD}$$

(24)

Note that $t_{FD}$ is a small number, which means that $t_{FD}<1$. Finally, we can derive a formula for $n$:

$$n>\frac{\log t_{GD}}{\log t_{FD}}$$

(25)

Thereby, we get a sufficient condition for $n$, and we can come to a conclusion. Gradient disappearance occurs in layers deep enough after feature disappearance. □

The above corollary is consistent with intuition. The disappearance of gradients is always accompanied by the disappearance of features, and it is always a problem in deep neural networks.

Theorem 4.

In neural networks, for any two tensors  $X_1$, $X_2$ of the same dimension, there are always two matrices  $M_1$, $M_2$, so that the operation of concating them and the operation of adding them after they have been multiplied in the Hadamard format by the two matrices, respectively, are equivalent in effect.

Proof. 

First of all, we illustrate the definition of the Hadamard product [26]. The Hadamard product (also known as the element-wise product) is a binary operation that takes two matrices of the same dimensions and produces another matrix of the same dimension. Concretely, we can define it as

$$\begin{array}{cc}\mathrm{A}\left(R,N\right)\circ \mathrm{B}& \left(R,N\right)=\mathrm{AB}\left(R,N\right)\\ \mathrm{AB}& {}_{ij}={\mathrm{A}}_{ij}{\mathrm{B}}_{ij}\end{array}$$

(26)

The symbol ‘$\circ$’ is used to distinguish it from the more common matrix product, which is denoted as ‘$\cdot$’ and is usually omitted. The definition implies that the dimension of $X_1$ should be the same as that of $M_1$, as well as $X_2$ and $M_2$. At the same time $X_1$ and $X_2$ are assumed to have the same dimensions in the precondition of our proposition. As such, we might as well set them to ${ℝ}^{R\times N}$. The representation of $X_1$ and $X_2$ is presented below:

$$X_1=\left(\begin{array}{c}{p}_1\\ p_2\\ ⋮\\ p_R\end{array}\right) X_2=\left(\begin{array}{c}{q}_1\\ q_2\\ ⋮\\ q_R\end{array}\right)$$

(27)

Our goal is to weigh the effect of the two operations. For the convenience of comparison, we let the results after the two operations multiply a matrix, thus converting the dimension to ${ℝ}^{R\times M}$. Adding a linear layer is very common in neural networks, and it hardly affects the network’s expression ability.

Considering the first scheme, the concat of $X_1$ and $X_2$, we have

$$\begin{array}{ll}U& =\left[X_1,X_2\right]\cdot A\\ & =\left(\begin{array}{cc}{p}_1& q_1\\ p_2& q_2\\ ⋮& ⋮\\ p_R& q_R\end{array}\right)\left(\begin{array}{cccc}{a}_1& a_2& \dots & a_M\end{array}\right)\end{array}$$

(28)

where $A\in {ℝ}^{2N\times M}$ and $U\in {ℝ}^{R\times M}$. Observing the $i$-th row and $j$-th column of $U$, we find that

$$\begin{array}{ll}{U}_{ij}& =\left[p_i,q_i\right]\cdot a_j\\ & =\left[p_i,q_i\right]\cdot \left[a_{jp},a_{jq}\right]\\ & =p_i\cdot a_{jp}+q_i\cdot a_{jq}\end{array}$$

(29)

Considering the second scheme, with the addition of $X_1$ and $X_2$ as the core, we have

$$\begin{array}{ll}V& =\left(M_1\circ X_1+M_2\circ X_2\right)\cdot B\\ & =\left(\begin{array}{c}{\lambda }_{11}\circ p_1+{\lambda }_{12}\circ q_1\\ {\lambda }_{21}\circ p_2+{\lambda }_{22}\circ q_2\\ ⋮\\ {\lambda }_{R1}\circ p_R+{\lambda }_{R2}\circ q_R\end{array}\right)\left(\begin{array}{cccc}{b}_1& b_2& \dots & b_M\end{array}\right)\end{array}$$

(30)

where $B\in {ℝ}^{N\times M}$ and $V\in {ℝ}^{R\times M}$. Still, we pay attention to the $i$-th row and $j$-th column of $V$ and find that

$$\begin{array}{ll}{V}_{ij}& =\left({\lambda }_{i1}\circ p_i+{\lambda }_{i2}\circ q_i\right)\cdot b_j\\ & =\sum \left(\left({\lambda }_{i1}\circ p_i+{\lambda }_{i2}\circ q_i\right)\circ b_j^{\mathrm{T}}\right)\\ & =\sum \left({\lambda }_{i1}\circ b_j^{\mathrm{T}}\circ p_i+{\lambda }_{i2}\circ b_j^{\mathrm{T}}\circ q_i\right)\\ & =\sum \left({\lambda }_{i1}\circ b_j^{\mathrm{T}}\circ p_i\right)+\sum \left({\lambda }_{i2}\circ b_j^{\mathrm{T}}\circ q_i\right)\\ & =p_i\cdot {\left({\lambda }_{i1}\circ b_j^{\mathrm{T}}\right)}^{\mathrm{T}}+q_i\cdot {\left({\lambda }_{i2}\circ b_j^{\mathrm{T}}\right)}^{\mathrm{T}}\\ & =p_i\cdot \left({\lambda }_{i1}^{\mathrm{T}}\circ b_j\right)+q_i\cdot \left({\lambda }_{i2}^{\mathrm{T}}\circ b_j\right)\end{array}$$

(31)

Comparing Equations (29) and (31), we find that, when we let ${\lambda }_{i1}^T\circ b_j$ equal $a_{jp}$ and ${\lambda }_{i2}^T\circ b_j$ equal $a_{jq}$, the values of $U$ and $V$ are equal, which is strong evidence of effect equivalence. □

As the equivalence has been proven, similar to the plain concat, no information is lost in the above method. We point out that the Hadamard product is an alternative version of the gate mechanism [27]. We use coefficients to adjust the original distribution to screen out effective features. For the speed and stability of training, setting the initial value of $M$ to 1 is recommended.

Further, we can observe the gradient of the parameters $\lambda$ in Equation (30), where we have

$$\nabla \left(\begin{array}{c}{\lambda }_{11}\circ p_1+{\lambda }_{12}\circ q_1\\ {\lambda }_{21}\circ p_2+{\lambda }_{22}\circ q_2\\ ⋮\\ {\lambda }_{R1}\circ p_R+{\lambda }_{R2}\circ q_R\end{array}\right)=\left(\begin{array}{cc}{p}_1& q_1\\ p_2& q_2\\ ⋮& ⋮\\ p_R& q_R\end{array}\right)$$

(32)

Compared to the gate mechanism, our method is simpler, saves space, and is more direct in gradient propagation.

Of course, Theorem 4 could be generalized to cases with an arbitrary number of tensors. We describe it in the following corollary:

Corollary 3.

In neural networks, for any tensors  $X_1$, $X_2$, $\dots$, $X_n$ of the same dimension, there always exist  $n$ matrices  $M_1$, $M_2$, $\dots$, $M_n$ so that the operation of concating them and the operation of adding them after they have been multiplied in the Hadamard format by the  $n$ matrices, respectively, are equivalent in effect.

Proof. 

This proof is similar to the proof for Theorem 4. □

Theorem 5.

In neural networks, for a layer composed of  $n$ neurons, the effective training times of the neurons in this layer reach the maximum when the dropout rate is set to  $0$ or  $1-\frac{1}{n}$.

Proof. 

The number of neurons in this layer is $n$, so we shall mark them as $N_1$, $N_2$, $\dots$, $N_n$. Suppose that the dropout rate [28] is $p$, and the total number of training times is $t$. We denote $1-p$ as q.

Consider the $t_i$-th training. The network randomly selects $nq$ neurons to update due to the existence of the dropout mechanism. Denote these neurons as $N_1$, $N_2$, $\dots$, $N_{nq}$.

Without the loss of generality, we consider the next time $N_1$ is selected, which is the $t_2$-th training time. We denote the number of neurons selected for update in $N_2$, $\dots$, $N_{nq}$ as $S$, and the number of neurons selected in $N_{nq+1}$, $\dots$, $N_n$ as $T$. We know that the selection of neurons in $S$ is an independent event, so we have

$$E\left(S\right)=q\left(nq-1\right)$$

(33)

At the same time, the relationship between $S$ and $T$ is

$$T=nq-1-S$$

(34)

Inferring from Equations (33) and (34), we achieve

$$E\left(T\right)=-n{q}^2+nq+q-1$$

(35)

The neurons represented by $S$ are the neurons that are updated jointly at time $t_2$ and time $t_1$, thus belonging to the same subnetwork. We assume that they share one training gain with $N_1$. At the same time, the neurons represented by $T$ have not been updated at time $t_1$; thus, each of them has one unique training gain. Therefore, at the update time $t_2$, the expected gain of $N_1$ is $1\times \frac{1}{E\left(T\right)+1}\times \frac{1}{E\left(S\right)+1}$, which is derived from the above proportion analysis. Paying attention to $t_1$ and $t_2$, we find that $t_2-t_1$ obeys geometric distribution because the selection of $N_1$ is a Bernoulli experiment with probability $q$. That is, $t_2-t_1\sim \mathcal{G}\mathcal{E}\left(q\right)$, meaning that

$$E\left(t_2-t_1\right)=\frac{1}{q}$$

(36)

Therefore, the expected number of training times for $N_1$ is $E\left(\frac{t}{t_2-t_1}\right)=tq$. The total training gain is the product of the number of training times and the gain of a single time training, which we denote as $G$. Now, the formula emerges:

$$G\left(q\right)=\frac{t}{-n^2{q}^3+\left(n^2+2n\right)q^2-\left(2n+1\right)q+n+1}$$

(37)

Denote $f\left(q\right)$ as the denominator of $G\left(q\right)$ and differentiate that to obtain

$$\frac{\partial f\left(q\right)}{\partial q}=-\left(nq-1\right)\left(3nq-2n-1\right)$$

(38)

With the help of Equation (38), it is easy to draw an image of $G\left(p\right)$, shown in Figure 1, where we set $t$ to $1$. The observation is that when $q=\frac{1}{n}$ or $q=1$, that is, $p$ is 0 or $1-\frac{1}{n}$, $G$ reaches the maximum value $\frac{t}{n}$, demonstrating that the effective training times of $N_1$ are the largest. The conclusion can be generalized to every neuron in the layer. □

Corollary 4.

In neural networks, if the amount of training data is sufficient, the optimal value of the dropout rate is 0.5; if the amount of training data is insufficient, then a number that is close to 1 is a better choice.

Proof. 

Theorem 5 focuses on the effective neuron training times in the network, and the corollary focuses on the representation ability. It can be seen from Equation (37) that the effective training times of a certain layer are directly proportional to the total training times $t$. When the number of training times reaches a certain threshold, the network reaches a balance point, and further training will not bring any performance improvements.

If the training data are sufficient, meaning that $t$ and $G$ are large enough, then the network is guaranteed to be fully trained. Therefore, we do not need to worry about whether the training times of neurons in the network is enough. However, we still need to consider the representation ability of the network, which has a close relationship with the number of subnetworks $SN$. It can be calculated as

$$SN=\left(\begin{array}{c}n\\ n\left(1-p\right)\end{array}\right)$$

(39)

which is a combination number. Obviously, when $p$ is $0.5$, the number of subnetworks is the largest, and the network’s representation ability is relatively strong.

However, when there are not enough training data, we cannot guarantee the sufficiency of training. On the one hand, we need to set the dropout rate to a value close to $0$ or $1-\frac{1}{n}$ to guarantee the number of trainings indicated by the theorem. On the other hand, in order to ensure the network’s representation ability, we want the dropout rate to be close to $0.5$. Here, a balanced approach is to choose the turning point shown in Figure 1, which considers both training times and representation ability. Because this point is difficult to analyze, we provide a fitting function shown in Figure 1, the error of which is bounded by $2\times {10}^{-2}$ for $n$ smaller than $512$. □

The above corollary is intuitive because the complexity of the network should be proportional to the amount of data. A small amount of data requires a simple model, calling for a higher dropout rate. Notice that a large dropout rate not only enables the model to be fully trained, but it also helps to accelerate the process.

In a modern neural network framework, the discarded neurons will not participate in gradient propagation this time, which largely reduces the number of parameters that need to be adjusted in the network.

3. Models

To address the task of chromatin accessibility prediction, we designed SemanticCAP, which includes a DNA language model that is shown in Section 3.1 and a chromatin accessibility model that is shown in Section 3.2. Briefly, we augment the chromatin accessibility model with the features provided by the DNA language model, thereby improving the chromatin accessibility prediction performance. A detailed methodology is described as follows.

3.1. DNA Language Model

3.1.1. Process of Data

We used the human reference genome GRCh37 (hg19) as the original data for our DNA language model. The human reference genome is a digital nucleic acid sequence database that can be used as a representative example of the gene set of an idealized individual of a species [29]. Therefore, a model based on the database could be applied to various genetic-sequence-related issues.

The task we designed for our DNA language model uses context to predict the intermediate base. However, there are at least three challenges. The first is that there are two inputs, the upstream and downstream, which are defined as the upper and lower sequences of a certain base. Since we predict the middle base from the information on both sides, the definition of the upstream and downstream are interchangeable, which means that the context should be treated in the same way. Second, the length of the input sequence is quite long, far from the output length of $4$ bp, which stands for the classification results of bases. The large gap between the input and output lengths points to the fact that neural networks must be designed in a more subtle way. Otherwise, redundant calculations or poor results may occur. Third, we do not always have such long context data in real situations. For example, the length of the upstreams in the DNA datasets in

Table 1. An overall view of accessible DNA segments of each cell. l_mean and l_med show the approximate length of the data, and l_std describes how discrete the data is. The length is denoted as bp.

Cell Type	Code	Size	l_min	l_max	l_mean	l_med	l_std
GM12878	ENCSR0000EMT	244,692	36	11,481	610	381	614
K562	ENCSR0000EPC	418,624	36	13,307	675	423	671
MCF-7	ENCSR0000EPH	503,816	36	12,041	471	361	391
HeLa-S3	ENCSR0000ENO	264,264	36	11,557	615	420	524
H1-hESC	ENCSR0000EMU	266,868	36	7795	430	320	347
HepG2	ENCSR0000ENP	283,148	36	14,425	652	406	626

mostly vary from $0$ bp to $600$ bp, resulting in insufficient information in some cases.

To solve the above problems, we designed a simple but effective input format and training method. First of all, we randomly selected a certain position, taking the upstream and downstream sequences with lengths of $512$ bp as the input, and the output is the base connecting the upstream and downstream, i.e., $\mathrm{A}$, $\mathrm{T}$, $\mathrm{C}$, and $\mathrm{G}$.

For the first challenge, we combined the upstream and downstream into one sequence, separated by a special prediction token $\left[\mathrm{LOST}\right]$, and provided different segment embeddings for the two parts. Additionally, a special classification token $\left[\mathrm{CLS}\right]$ was added to the beginning of the sequence so that the model could learn an overall representation of it. For the second challenge, the final hidden state corresponding to the token $\left[\mathrm{LOST}\right]$ was used as the aggregate sequence representation for classification tasks, by which the output dimension was reduced quickly without complex network structures. This technique was used by Bert [30] for the first time. For the third challenge, some data augmentation tricks were applied to enhance the capabilities of the model. First, we constructed symmetric sequences based on the principle of base complementation and the non-directionality of DNA sequences, including the axial symmetry and mirror symmetry. This helps the model learn the two properties of DNA sequences. Second, we did not include all of the inputs in the model, which helps to enhance the model’s prediction ability under conditions with insufficient information. Basically, we mask some percentage of the input tokens at random, and concretely there are two strategies. For a certain sequence, either upstream or downstream, we mask (replace with $\left[\mathrm{MASK}\right]$) $20\%$ of random tokens in $10\%$ of cases or $40\%$ of consecutive tokens in $15\%$ of cases. Figure 2 is an example of our mask operation. In this case, two random tokens of the upstream and three consecutive tokens of the downstream are masked.

Finally, there is no need to worry about overfitting. First, we have 10⁹ bases in the DNA dataset, meaning that we will not over-learn some specific data. Second, we have mask and dropout operations in our training, which both are great ways to avoid over-training.

3.1.2. Model Structure

The input and output are constructed as described in Section 3.1.1, and we denote them as $T_{in}$ and $T_{out}$. Basically, the model can be described as

$$\begin{array}{ll}{T}_{in}& \stackrel{\mathrm{embed}}{\to }{T}_{embed}\stackrel{\mathrm{multi}-\mathrm{conv}}{\to }{T}_{cnns}\\ & \stackrel{\mathrm{transformer}}{\to }{T}_{trans}\stackrel{\mathrm{mlp}}{\to }{T}_{out}\end{array}$$

(40)

where $\mathrm{embed}$ is the input embedding layer, $\mathrm{multi}-\mathrm{conv}$ stands for our multi-kernel CNN, $\mathrm{transformer}$ represents the transformer blocks, and $\mathrm{mlp}$ contains a linear layer and a softmax function. Figure 3 shows the full picture of the model.

Function $\mathrm{embed}()$ is the encoding layer transforming the input sequence into a matrix. The dimension conversion is ${\mathbb{N}}^L\stackrel{}{\to }{ℝ}^{L\times E}$, where $L$ is the length of the sequence, and $E$ is the encoding length. Specifically, we encode the input as

$$\begin{array}{ll}\mathrm{embed}\left(T_{in}\right)=& \mathrm{word}-\mathrm{embed}\left(T_{in}\right)\\ & +\mathrm{position}-\mathrm{embed}\left(T_{in}\right)\\ & +\mathrm{segment}-\mathrm{embed}\left(T_{in}\right)\end{array}$$

(41)

where $\mathrm{word}-\mathrm{embed}()$ is the meaning of the word itself, $\mathrm{position}-\mathrm{embed}()$ provides the representation of different positions, and $\mathrm{segment}-\mathrm{embed}()$ distinguishes the upstream and the downstream. An intuitive approach is to concatenate the three encodings without losing semantics, but this requires triple the space. Instead, we directly add these three encodings. This works because the three parameters are all leaf nodes of the training graph and can automatically adapt to each other’s distributions. In this way, we reduce the dimension of the coded matrix, thus reducing the parameter space and data space.

Function $\mathrm{multi}-\mathrm{conv}()$ is the multi-kernel convolution layer learning a short-range relationship of the sequence. The dimension conversion is ${ℝ}^{L\times E}\stackrel{}{\to }{ℝ}^{L\times H}$, where $H$ is the hidden dimension. Here, we use convolution kernels of different lengths to learn local relationships at different distances, and we propose a smooth feature addition (SFA) method to fuse these features. Specifically, we carry out

$$\mathrm{multi}-\mathrm{conv}\left(T_{embed}\right)={\displaystyle \sum }_{i=0}^k{\lambda }_i\circ \mathrm{LN}\left({\mathrm{conv}}_i\left(T_{embed}\right)\right)$$

(42)

where ${\mathrm{conv}}_i()$ is a normal, one-dimensional, convolution layer with a kernel length of $l_i$, the output dimension of which is ${ℝ}^{L\times H}$, ${\lambda }_i$ is a network parameter with a dimension of ${ℝ}^{L\times H}$, and $k$ is the number of kernels of different lengths. The sizes of the convolution kernels are small rather than large, and their advantages have been verified in DenseNet [31]. On the one hand, small convolution kernels use less space than large convolution kernels. On the other hand, we need small convolution kernels to learn the local information of the sequence, while the long-range dependence of the sequence is to be explored by the subsequent $\mathrm{transformer}$ module.

Now, we will explain how we designed the smooth feature addition (SFA) algorithm. Before that, we must provide insight into what happens in the plain concat of features.

In a sequence problem, we often directly concat two features in the last dimension. Specifically, if we have two features with dimensions ${ℝ}^{L\times M}$ and ${ℝ}^{L\times N}$, the dimension of the features after concat is ${ℝ}^{L\times \left(M+N\right)}$. We thought that this approach would not lose information, but, in fact, there is a danger of feature disappearance. For two features with different distributions learning from different modules, plain concat will create an unbalanced distribution, where some values are extremely small. To make matters worse, layer normalization is usually used to adjust the distribution after a concat operation, causing the values to be concentrated near $0$. Quantitative analysis can be seen in Theorem 3. Finally, as the network goes deeper, the gradient disappears, leading to the difficulty of learning. This is proven in Corollary 2.

A naive thought is to normalize the two distributions before concating them, which is proven to be correct in Theorem 1. However, it is not effective, for it converts the dimension from ${ℝ}^{L\times H}$ to ${ℝ}^{L\times kH}$, posing a challenge for the subsequent module design. Considering that the dimensions of convolution features are the same, this inspired us to find a way to smoothly add them using some tuning parameters. This is how we designed SFA. Corollary 3 proves the equivalence of SFA and plain concat, and it illustrates the working mechanism of SFA and its advantages in space occupation, feature selection, and gradient propagation.

Function $\mathrm{transformer}()$ is the stack of transformer blocks learning a long-range relationship of the sequence. The dimension conversion is ${ℝ}^{L\times H}\stackrel{}{\to }{ℝ}^{L\times H}$. Simply, it can be described as

$$\mathrm{transformer}\left(T_{cnns}\right)=\mathrm{sub}\left(\mathrm{ff},\mathrm{sub}\left(\mathrm{attention},T_{cnns}\right)\right)$$

(43)

where $\mathrm{sub}\left(f,x\right)=\mathrm{LN}\left(x+f\left(x\right)\right)$. $\mathrm{ff}$ represents the feed forward function, and $\mathrm{attention}$ is short for multi-head attention. The module was proposed by Vaswani et al. in 2017 [23].

Function $\mathrm{mlp}()$ is the output layer and is responsible for converting the hidden state to the output. The dimension conversion is ${ℝ}^{L\times H}\stackrel{}{\to }\mathbb{N}$. We extract the tensor corresponding to the token $\left[\mathrm{LOST}\right]$, convert it into an output probability through a linear layer, and generate the prediction value via a softmax function. The output process is

$$\mathrm{mlp}\left(T_{trans}\right)=\mathrm{softmax}\left(\mathrm{linear}\left(T_{trans}\left[{}^{\prime }\left[\mathrm{LOST}\right]{}^{\prime }\right]\right)\right)$$

(44)

3.2. Chromatin Accessibility Model

3.2.1. Process of Data

We selected DNase-seq experiment data from six typical cell lines, including GM12878, K562, MCF-7, HeLa-S3, H1-hESC, and HepG2, as the original data for our chromatin accessibility model. GM12878 is a type of lymphoblast produced by EBV transformation from the blood of a female donor of Northern European and Western European descent.

K562 is an immortalized cell derived from a female patient with chronic myeloid leukemia (CML). MCF-7 is a breast cancer cell sampled from a white female. HeLa-S3 is an immortal cell derived from a cervical cancer patient. H1-hESC is a human embryonic stem cell. HepG2 comes from a male liver cancer patient.

For each cell type, we downloaded the original sequence data from the ENCODE website, used a short read aligner tool bowtie [32] to map the DNA sequence to the human reference genome (hg19), and used HOTSPOT [33] to identify chromatin accessibility regions (peaks), i.e., genome-wide open chromatin regions that can yield information about possible protein binding regions on a genome-wide scale. We treated these variable-length sequences as positive samples. At the same time, we sampled the same number and same size sequences from the whole genome as negative samples. An overview of the data is shown in Table 1, which shows the number of sequences, the minimum value, the median value, the maximum value, and the standard deviation in lengths. Additionally, the distribution statistics of different datasets are shown in Figure 4. For the fairness of comparison, we removed sequences with lengths of less than $36$ bp. We truncated or expanded each sequence symmetrically to a sequence of length $768$ bp, and we took a context of a length of $512$ bp for each site in it. Therefore, the actual input length of our model is $768+512\times 2=1792$ bp. From Figure 4, we can observe that most of the lengths are clustered between 36 and 1792. This proves that our cut-off has little impact and is reasonable. Similar to our DNA language model, a special classification token $\left[\mathrm{CLS}\right]$ was added to the beginning of the sequence to predict the accessibility. Compared to the input length of $800$ bp in [15], our prediction length increased by 124%, and the quantity of the DNA sequences that did not need to be truncated in the original dataset increased by 17.4%. Moreover, we did not pay a great price for such a long input because our context was transferred to a pre-trained model for the predictions. The output is the accessibility of the input sequence, i.e., either $0$ for inaccessibility or $1$ for accessibility.

Finally, the ratio of our training set, validation set, and test set is $0.85:0.05:0.10$. The training set was used to train the model, the validation set was used to adjust the hyperparameters to prevent overfitting, and the test set was used to test the performance of the final model.

3.2.2. Model Structure

The input and output were constructed as described in Section 3.2.1. and are denoted as $T_{in}$ and $T_{out}$. Basically, the model can be described as

$$\begin{array}{ll}& T_{in}\stackrel{\mathrm{embed}}{\to }{T}_{embed}\stackrel{\mathrm{multi}-\mathrm{conv}}{\to }{T}_{cnns}\stackrel{\mathrm{sconcat}}{\to }{T}_{sconcat}\\ & \stackrel{\mathrm{transformer}}{\to }{T}_{trans}\stackrel{\mathrm{mlp}}{\to }{T}_{out}\end{array}$$

(45)

where $\mathrm{embed}$ is the input embedding layer, $\mathrm{multi}-\mathrm{conv}$ stands for our multi-kernel CNN, $\mathrm{sconcat}$ is short for our SConcat module, $\mathrm{transformer}$ represents the transformer blocks, and $\mathrm{mlp}$ contains a linear layer and a sigmoid function. Figure 5 shows a full picture of the model. One may find that the accessibility model is very similar to our DNA language model. Indeed, we only modified some of the model structures and changed the hyperparameters, but they are all very critical adjustments that make the model suitable for the task.

Function $\mathrm{embed}()$ is the encoding layer transforming the input sequence into a feature matrix. The dimension conversion is ${\mathbb{N}}^L\stackrel{}{\to }{ℝ}^{L\times E}$, where $L$ is the length of the sequence, and $E$ is the encoding length. Specifically, we encode the input as

$$\begin{array}{ll}\mathrm{embed}\left(T_{in}\right)=& \mathrm{word}-\mathrm{embed}\left(T_{in}\right)\\ & +\mathrm{position}-\mathrm{embed}\left(T_{in}\right)\end{array}$$

(46)

Note that there is no $\mathrm{segment}-\mathrm{embed}()$ in this task because there is no need to distinguish between the different segments.

Function $\mathrm{multi}-\mathrm{conv}()$ has been explained in Section 3.1.2. The dimension conversion is ${ℝ}^{L\times E}\stackrel{}{\to }{ℝ}^{L\times G}$, where $G$ is the dimension of features learning from this layer.

Function $\mathrm{sconcat}()$ is the concat layer that fuses the features of the language model with the features learned from $\mathrm{multi}-\mathrm{conv}$. The dimension conversion is ${ℝ}^{L\times G}\stackrel{}{\to }{ℝ}^{L\times \left(G+H\right)}$, where $H$ is the dimension of features generated from the DNA language model. Basically, the language model was used to construct features for different sites in the sequence, and a smooth feature concat (SFC) method was proposed to fuse them with the previous features:

$$\mathrm{sconcat}\left(T_{in},T_{cnns}\right)=\mathrm{LN}\left(\left[{\lambda }_1\circ \mathrm{LM}\left(\overleftrightarrow{T_{in}}\right),{\lambda }_2\circ T_{cnns}\right]\right)$$

(47)

where $\overleftrightarrow{T_{in}}$ stands for the context of sites in $T_{in}$; ${\lambda }_1$ and ${\lambda }_2$ are two network parameters with a dimension of ${ℝ}^L$; and $\mathrm{LM}$ refers to our DNA language model. Here, it receives a DNA sequence, then constructs the context for each site in the sequence, and produces an output of length $H$. Specifically, if the length of the sequence is $L$, it will construct $L$ pairs of contexts as the input and output an ${ℝ}^{L\times H}$ matrix.

Now, we explain how we designed the smooth feature concat (SFC) algorithm. First, we should mention that the output dimension of the language model is ${ℝ}^{L\times H}$, and the dimension of $T_{cnns}$ is ${ℝ}^{L\times G}$, which means we cannot directly apply SFA in this scenario.

Fortunately, the analysis in Section 3.1.2 has already provided a solution to this problem. We can normalize the two distributions separately before concating them. However, this method uses $\mathrm{LN}$ twice and consumes additional parameter space and data space. One question is whether it is possible to use $\mathrm{LN}$ only once. It appears that this is the case. Theorem 2 states that, for any two distributions, there always exist two coefficients, so that the concat after they are multiplied by these two coefficients is a standardized distribution. That is how our SFA works. We multiply the two tensors by two coefficients, and we then carry out layer normalization after their concatenation. As such, we fused the two features smoothly with only one use of the $\mathrm{LN}$ operation. Interestingly, this method is a weakened version of Theorem 4.

Function $\mathrm{transformer}()$ is the same as that described in Section 3.1.2. The dimension conversion is ${ℝ}^{L\times F}\stackrel{}{\to }{ℝ}^{L\times F}$, where $F=G+H$.

Function $\mathrm{mlp}()$ is the output layer and is responsible for transforming the hidden state to the output. The dimension conversion is ${ℝ}^{L\times F}\stackrel{}{\to }\mathbb{N}$. We extracted the tensor corresponding to the token $\left[\mathrm{CLS}\right]$, converted it into an output probability through a linear layer, and generated the prediction value via a sigmoid function. The output process is

$$\mathrm{mlp}\left(T_{trans}\right)=\mathrm{sigmoid}\left(\mathrm{linear}\left(T_{trans}\left[{}^{\prime }\left[\mathrm{CLS}\right]{}^{\prime }\right]\right)\right)$$

(48)

4. Results and Discussions

4.1. Semantic DNA Evaluation

We compared the performance of our proposed method with several baseline methods, including the gapped k-mer SVM (gkm-SVM) [9], DeepSEA [14], and k-mer [15] methods. For the sake of fairness, all of the parameters were set as defaults. Moreover, to prove the effectiveness of the DNA language model, we also tested our accessibility model after excluding the DNA language model. For evaluation purposes, we computed two often-used measures, the area under the receiver operating characteristic curve (auROC) and the area under the precision-recall curve (auPRC), which are good indicators of the robustness of a prediction model. The classification results for six datasets are shown in

Table 2. The results of the comparative experience to test the chromatin accessibility prediction system. Refer to Table 1 for the codenames of these datasets.

System	MT	PC	PH	NO	MU	NP	Average
(a) auROC
Gkm-SVM	0.8528	0.8203	0.8967	0.8648	0.8983	0.8359	0.8697
DeepSEA	0.8788	0.8629	0.9200	0.8903	0.8827	0.8609	0.8782
k-mer	0.8830	0.8809	0.9212	0.9016	0.9097	0.8722	0.8975
no feature 1	0.8727	0.8664	0.9058	0.8840	0.8849	0.8699	0.8806
SemanticCAP	0.8907	0.8883	0.9241	0.9001	0.8982	0.8847	0.8977
(b) auPRC
Gkm-SVM	0.8442	0.8081	0.8860	0.8627	0.8823	0.8123	0.8504
DeepSEA	0.8758	0.8551	0.9146	0.8888	0.8705	0.8508	0.8801
k-mer	0.8774	0.8732	0.9156	0.8992	0.8968	0.8630	0.8973
no feature 1	0.8745	0.8663	0.9053	0.8852	0.8878	0.8730	0.8820
SemanticCAP	0.8914	0.8896	0.9218	0.9004	0.8993	0.8871	0.8983

¹ no feature is SemanticCAP without pre-trained features.

. Compared to the best baseline k-mer, our system shows a maximum improvement of 1.25% in auROC, and a maximum improvement of 2.41% in auPRC. Although some results on some datasets are not good, our model outperforms k-mer on average, with a 0.02% higher auROC score and a 0.1% higher auPRC score. Compared to gkm-SVM and DeepSEA, SemanticCAP shows an average improvement of about 2–3%. Finally, the introduction of our DNA language model resulted in performance improvements of 2%.

We also tested the accessibility prediction accuracy of the loci shared in different cell lines. For example, GM12878 and HeLa-S3 have 20 common loci, and the prediction accuracy of these 20 loci in both cell lines is 85% and 90%, respectively. Another example is that K562 and MCF-7 have 21 common loci, and the prediction accuracy is 80.9% and 90.5%, respectively. This shows the applicability of our system on the common loci between different cell lines.

4.2. Analysis of Models

4.2.1. Effectiveness of Our DNA Language Model

We performed experiments on several different DNA language model structures, which can be divided roughly into two categories. The first category can be attributed to methods based on normal CNNs, and the second category uses our multi-conv architecture with data augmentation. Six structures were tested. At the same time, in order to test the prediction ability of different models in the case of insufficient information, we randomly masked some words and tested the results. The complete results are shown in

Table 3. The results of the experiment comparing DNA language models.

Model	Loss (No Mask)	Accuracy (No Mask)	Accuracy (Mask 30%)
convs (max) + lstms	1.152	0.4538	0.3265
convs (max) + attention (ReLU)	1.113	0.4814	0.3687
convs (avg) + trans 1	1.097	0.4926	0.3599
mconv 2 (PC 3) + trans	0.968	0.5114	0.4572
mconv (PA 3) + trans	0.931	0.5187	0.4784
mconv (SFA 3) + trans	0.921	0.5202	0.4793

¹ trans refers to transformer+linear. ² mconv stands for our multi-conv layer. ³ PA is plain add, PC is plain concat, and SFA is our smooth feature addition method.

. Through the comparison of LSTM and Attention, we found that the attention mechanism can greatly improve the prediction ability of the DNA language model. When using the MaxPooling and ReLU functions, we observed that the output of the last hidden layer was mostly $0$, where the number of effective (not zero) neurons is about 3/192. This happens because the ReLU function shields neurons whose values are less than $0$, and MaxPooling selectively updates specific neurons. Therefore, we replaced MaxPooling with AveragePooling, and the Attention layer that uses the ReLU function was replaced with a transformer. That is the third method listed in Table 3. The second category uses multi-conv to extract the local features of the sequence. The introduction to the multi-conv mechanism with data augmentation strategies brought increases in accuracy, especially when some tokens were masked. There are three kinds of feature fusion strategies: plain concat (PC), plain add (PA), and our smooth feature add (SFA). The third, fourth, and fifth items in the table indicate that SFA outperforms the other two fusion methods. The last item in Table 3, mconv(SFA)+trans, is the model that we finally chose as our DNA language model.

4.2.2. Effectiveness of Our Chromatin Accessibility Model

We experimented with several chromatin accessibility model structures, all of which were based on the transformer. The main difference is the use of multi-conv and the modules after the transformer. A complete comparison of the results is shown in

Table 4. The results of the experiment comparing the chromatin accessibility models.

Model	Parameters (M)	Total (h)	auROC	auPRC	F1
PC + trans 1 + lstm	4.16	4.6	0.8595	0.8625	0.7880
PC + trans + conv + lstm	4.95	2.9	0.8741	0.8765	0.8036
PC + trans + flatten	16.4	2.0	0.8822	0.8839	0.8124
PC + trans + conv + flatten	6.13	2.8	0.8817	0.8834	0.8119
PC + trans + linear	3.84	1.5	0.8839	0.8854	0.8144
mconv + PC + trans + linear	5.61	2.5	0.8881	0.8902	0.8590
mconv + SFC 2 + trans + linear	5.61	2.5	0.8907	0.8914	0.8606

¹ trans is short for transformer blocks. ² SFC is our smooth feature concat method.

.

First, we focused on the module before the transformer. We noticed that the introduction of multi-conv also resulted in performance improvements, especially in F1. In our chromatin accessibility model, we concatenated the features provided by the DNA language model, where we could either directly concat (PC) them or use our SFC method. The evaluation values of the last two items show the superiority of SFC.

Now, we will turn to the comparison of modules after the transformer. The transformation from the features of the transformer to the result is a challenge. In this part, five methods were tested. It should be mentioned that mconv + SFC + trans + linear is the final model. In terms of training time, our model can be fully parallelized, making it more advantageous than LSTM, based on recurrent networks. At the same time, our model has fewer parameters and has a simpler structure than Flatten after CNNs and can thus converge quickly. In terms of evaluation, the LSTM-based methods performed poorly. The main reason for this is that it is difficult for LSTM to learn the long-range dependence of a sequence. The convolution layer improves the performance of the LSTM to some extent by shortening the sequence length. In methods that are based on Flatten, introducing convolution layers actually reduces the accuracy. This could be caused by the convolution layers destroying the sequence features learned from the transformer. During multiple chromatin accessibility models, the method using multi-conv and our smoother concat (SFC) method obtained the best results with a relatively small number of parameters.

4.3. Analysis of [CLS]

We were able to observe the effectiveness of introducing the $\left[\mathrm{CLS}\right]$ symbol into our accessibility model. A direct indicator is the feature corresponding to $\left[\mathrm{CLS}\right]$ after the transformer layer, i.e., the value of ${\mathrm{T}}_{\mathrm{trans}}\left[{}^{\prime }\left[\mathrm{CLS}\right]{}^{\prime }\right]$ in Equation (48). We randomly selected a certain number of positive and negative samples and used our chromatin accessibility model to predict them. For each sample, we output the $256$-dimensional tensor corresponding to $\left[\mathrm{CLS}\right]$ after the transformer layer, and we reduced it to 2-dimensional space with t-SNE, which is shown in Figure 6. According to the figure, the feature has the ability to distinguish positive and negative examples, which is strong evidence of its effectiveness.

4.4. Analysis of SFA and SFC

In this section, we conducted two comparison experiments of PA, PC, SFA, and SFC.

When testing the various DNA language models, we made a comparison between SFA, PA, and PC, which correspond to the last three items in Table 3. We used $5\times {10}^6$ samples to train the three models, drew a training loss map of them, and saw what would happen, which is shown in Figure 7a. PA quickly reduces losses at the fastest speed at the beginning because all of the features in multi-conv are trained to the same degree at the same time. However, in the later stage, there appears a phenomenon in which some features are overtrained while others are not, leading to the oscillation of loss.

In the experiment of various chromatin accessibility models, we made a comparison between SFC and PC, corresponding to the last two items in Table 4. The first $5\times {10}^3$ samples were used to measure its training state, which is shown in Figure 7b. As we can see, PC has a lower training speed because it has a problem regarding gradient disappearance. Compared to it, the gradient propagation of SFA is selective and more stable for the whole term.

We can observe the effectiveness of SFA from another angle. Paying attention to the parameters $C_{SFA}$ of SFA in multi-conv, whose dimension is ${ℝ}^{K\times L\times H}$, where $K$ is the number of kernels, $L$ is the sequence length, and $H$ is the hidden dimension, we normalized it, and converted it to ${C^{\prime }}_{SFA}$, whose dimension is ${ℝ}^{K\times L}$. This was carried out for both the language model and the chromatin accessibility model, and they can be observed in Figure 8. Note that the sum of the vertical axis in Figure 8a,b is always $1$ due to the normalization. Obviously, different sequence positions and different convolution kernels have different weights, which proves SFA’s ability to regulate features.

In general, SFA and SFC make training smoother, faster, and better than ordinary concatenation and addition. They are smoother because we used parameters to regulate features. They are faster because they speed up the training of the model by avoiding the gradient problem. They are simple but effective. In fact, since they share the same essence (the Hadamard product), they share the same advantages.

5. Conclusions

In this article, we propose a chromatin accessibility prediction model called SemanticCAP. Our model is able to predict open DNA regions, thus having a guiding role in disease detection, drug design, etc. For example, a gene called CYMC from cell H1-hESC mutated in the middle with a length of 5 bp, and its accessibility decreased from 0.98 to 0.14 as predicted by our model, which is consistent with the experimental data that it reduces transcription [34]. Another example is a mutation in a gene called HNF4A from cell K562, which leads to a reduction in gene expression [35]. Our model predicted that its accessibility decreased from 0.66 to 0.2, which provides a reasonable explanation for the experimental phenomena of reduction in gene expression caused by HNF4A mutation. Similarly, we can monitor the accessibility changes of DNA targeted by drugs (especially anticancer drugs), and the change of accessibility will provide guidance for drug action. Our main innovations are as follows. First, we introduced the concept of language models in natural language processing to model DNA sequences. This method not only provides the word vector presentation of the base itself, but it also provides sufficient information about the context of a site in a DNA sequence. Second, we used a small number of parameters to solve the feature fusion problem between different distributions. Specifically, we solve the problem of the smooth addition of distributions with the same dimensions using SFA and the problem of the smooth concatenation of distributions with different dimensions using SFC.

Third, we use an end-to-end model design, in which we fully utilize the learning ability and characteristics of the convolution and attention mechanism, thus achieving a better result with fewer parameters and a shorter training time.

Of course, there is still room for improvement in our method. In terms of the sample construction, we randomly selected the same number of DNA sequences with the same length as negative samples. This approach may be modified. For example, we could deliberately use an unbalanced dataset because there are so much DNA data, and we could then use some strategies, such as ensemble learning [36], to eliminate the negative effects of data imbalance [37]. In terms of data input, sequence truncation, and sequence completion operations exist in our model, which may cause information loss or redundant calculations. Additionally, the task we designed for the DNA language model could also be enhanced. Multiple positions can be predicted simultaneously, similar to the cloze problem in Bert. There are also some limitations in the current study. The first limitation is that the attention mechanism consumes too much memory, which could be replaced by a short-range attention or a mixed-length attention [38]. Additionally, our smooth feature fusion methods, SFA and SFC, could also be used in the multi-head attention to save space and accelerate training. Moreover, the dropout mechanism makes all neurons effective in the prediction phase, but there may exist a more reasonable way of fusing subnetworks. These issues need to be further explored.