Logistic Biplot by Conjugate Gradient Algorithms and Iterated SVD

Abstract

Multivariate binary data are increasingly frequent in practice. Although some adaptations of principal component analysis are used to reduce dimensionality for this kind of data, none of them provide a simultaneous representation of rows and columns (biplot). Recently, a technique named logistic biplot (LB) has been developed to represent the rows and columns of a binary data matrix simultaneously, even though the algorithm used to fit the parameters is too computationally demanding to be useful in the presence of sparsity or when the matrix is large. We propose the fitting of an LB model using nonlinear conjugate gradient (CG) or majorization–minimization (MM) algorithms, and a cross-validation procedure is introduced to select the hyperparameter that represents the number of dimensions in the model. A Monte Carlo study that considers scenarios with several sparsity levels and different dimensions of the binary data set shows that the procedure based on cross-validation is successful in the selection of the model for all algorithms studied. The comparison of the running times shows that the CG algorithm is more efficient in the presence of sparsity and when the matrix is not very large, while the performance of the MM algorithm is better when the binary matrix is balanced or large. As a complement to the proposed methods and to give practical support, a package has been written in the R language called BiplotML. To complete the study, real binary data on gene expression methylation are used to illustrate the proposed methods.

1. Introduction

In many studies, researchers have a binary multivariate data matrix and aim to reduce dimensions to investigate the structure of the data. For example, in the measurement of brand equity, a set of consumers evaluates the perceptions of quality, perceptions of value, or other brand attributes that can be represented in a binary matrix [1]; in the evaluation of the impact of public policies, the answers used to identify whether the beneficiaries have some characteristics or to identify if some economic or social conditions have changed from a baseline are usually binary [2,3,4]. Likewise, in biological research—and in particular in the analysis of genetic and epigenetic alterations—the amount of binary data has been increasing over time [5]. In these cases, classical methods to reduce dimensionality, such as principal component analysis (PCA), are not appropriate.

This problem has received considerable attention in the literature; consequently, different extensions of PCA have been proposed. From a probabilistic perspective, Collins et al. [6] provide a generalization of PCA to exponential family data using the generalized linear model framework. This approach suggests the possibility of having proper likelihood loss functions depending on the type of data.

Logistic PCA is the extension of the classic PCA method for binary data and was studied by Schein et al. [7] using the Bernoulli likelihood, where an alternating least squares method is used to estimate the parameters. De Leeuw [8] proposed the calculation of the maximum likelihood estimates of a PCA on the logit or probit scale, using a MM algorithm that iterates a sequence of weighted or unweighted singular value decompositions. Subsequently, Lee et al. [9] introduced sparsity to the loading vectors defined on the logit transform of the success probabilities of the binary observations and estimated the parameters using an iterative weighted least squares algorithm, but the algorithm is computationally too demanding to be useful when the data dimension is high. To solve this problem, a different method was proposed by Lee and Huang [10] using a combined algorithm with coordinate descent and MM to reduce the computational effort. More recently, Landgraf and Lee [11] proposed a formulation that does not require matrix factorization, and they use an MM algorithm to estimate the parameters of the logistic PCA model. Song et al. [12] proposed the fitting of a logistic PCA model using an MM algorithm across a non-convex singular value threshold to alleviate the overfitting issues. However, neither of these approaches provides a simultaneous representation of rows and columns to visualize the binary data set, similar to what is called a biplot for continuous data [13].

The biplot methods allow the simultaneous representation of the individuals and variables of a data matrix [14]. Biplots have proven to be very useful for analyzing multivariate continuous data [15,16,17,18,19] and have also been implemented to visualize the results of other multivariate techniques such as multidimensional scaling, MANOVA, canonical analysis, correspondence analysis, generalized bilinear models, and the HJ-Biplot, among many others [14,20,21,22,23,24].

In cases where the variables of the data matrix are not continuous, a classical linear biplot representation is not suitable. Gabriel [25] proposed the “bilinear regression” to fit a biplot for data with distributions from the exponential family, but the algorithm was not clearly established and was never used in practice. For multivariate binary data, Vicente-Villardon et al. [26] proposed a biplot called Logistic Biplot (LB), which is a dimension reduction technique that generalizes PCA to cope with binary variables and has the advantage of simultaneously representing individuals and variables. In the LB, each individual is represented by a point and each variable as directed vectors, and the orthogonal projection of each point onto these vectors predicts the expected probability that the characteristic occurs. The method is related to logistic regression in the same way that classical biplot analysis is related to linear regression. In the same way, as linear biplots are related to PCA, LB is related to Latent Trait Analysis (LTA) or Item Response Theory (IRT).

The authors estimate the parameters of the LB model by a Newton–Raphson algorithm, but this presents some problems in the presence of separation or sparsity. In [27], the method is extended using a combination of principal coordinates and standard logistic regression to approximate the LB model parameters in the genotype classification context and is called an external logistic biplot, but the procedure of estimation is quite inefficient for big data matrices. More recently, in [28], the external logistic biplot method was extended for mixed data types, but the estimation algorithm still has problems with big data matrices or in the presence of sparsity. Therefore, there is a clear need to extend the previous algorithms for the LB model because they are not very efficient and none of them present a procedure for choosing the number of dimensions of the final solution of the model.

In the context of supervised learning, some optimization methods have been successfully implemented for logistic regression. For example, Komarek and Moore [29] develops Truncated-Regularized Iteratively-Reweighted Least Squares (TR-IRLS) technique that implements a linear Conjugate Gradient (CG) to approximate the Newton direction. The algorithm is especially useful for large, sparse data sets because it is fast, accurate, robust to linear dependencies, and no data preprocessing is necessary. Furthermore, another of the advantages of the CG method is that it guarantees convergence in a maximum number of steps [30]. On the other hand, when the imbalance is extreme, this problem is known as the rare events problem or the imbalanced data problem, which presents several problems and challenges to existing classification algorithms [31]. Maalouf and Siddiqi [32] developed a method of Rare Event Weighted Logistic Regression (RE-WLR) for the classification of imbalanced data on large-scale data.

In this paper, we propose the estimation of the parameters of the LB model in two different ways: one of these is to use CG methods, and the other way is to use a coordinate descendent MM algorithm. In addition, we incorporate a cross-validation procedure to estimate the generalization error and thus choose the number of dimensions in the LB model.

Taking into account the latent variables and model specification that defines a LB, a simulation process is carried out that allows for the evaluation of the performance of the algorithms and their ability to identify the correct number of dimensions to represent the multivariate binary data matrix adequately. Besides the proposed methods, the BiplotML package [33] was written in R language [34] to give practical support to the new algorithms.

The paper is organized into the following sections. Section 2.1 presents the classical biplot for continuous data. Next, Section 2.2 presents the formulation of the LB model. Section 2.3 introduces the proposed adaptation of the CG algorithm and the coordinate descendent MM algorithm to fit the LB model. Section 2.4 describes the model selection procedure (number of dimensions) and introduces our formulation via simulated data. Section 3 presents the performance of the proposed models and an application using real data. Finally, Section 4 presents a discussion of the main results.

2. Materials and Methods

2.1. Biplot for Continuous Data

Classical biplot methods allow the combined visualization of the rows and columns of a data matrix $\mathbf{X}={\left({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\right)}^T$, with ${\mathbf{x}}_i\in {\mathbb{R}}^p$, $i=1,\dots ,n$ being a vector of the observations of an individual i on p variables in low-dimensional space [13,14].

More generally, if $rank\left(\mathbf{X}\right)=r$ and the data are centered, given a positive integer $k\le r$, a biplot is an approximation

$$\mathbf{X}={\mathbf{AB}}^T+\mathbf{E},$$

(1)

where $\mathbf{A}$ and $\mathbf{B}$ are matrices of rank k, and $\mathbf{E}$ is the matrix that contains the approximation errors. In this way, the matrix $\mathbf{X}$ can be graphically represented using markers (points or vectors) ${\mathbf{a}}_1,\dots ,{\mathbf{a}}_n$ for its rows and ${\mathbf{b}}_1,\dots ,{\mathbf{b}}_p$ for its columns, in such a way that the $ij$-th element of the matrix, $x_{ij}$, is approximated by the inner product ${\mathbf{a}}_i^T{\mathbf{b}}_j$, and the natural space of parameters is determined by $\mathbf{\Theta }=\mathbf{A}{\mathbf{B}}^T$.

It is well known that we can reproduce exactly the matrix in dimension r using its singular value decomposition (SVD),

$$\mathbf{X}=\mathbf{U}\Lambda {\mathbf{V}}^T,$$

(2)

where $\mathbf{U}=\left[{\mathbf{u}}_1,\dots ,{\mathbf{u}}_r\right]$ and $\mathbf{V}=\left[{\mathbf{v}}_1,\dots ,{\mathbf{v}}_r\right]$ are the matrices of left and right singular vectors and $\Lambda$ is the r-dimensional diagonal matrix containing the singular values in decreasing order ${\lambda }_1\ge \dots \ge {\lambda }_r>0$. It is also known that the low (k)-rank approximation for $\mathbf{X}$ is obtained from

$$\mathbf{\Theta }={\mathbf{U}}_{\left(k\right)}{\Lambda }_{\left(k\right)}{\mathbf{V}}_{\left(k\right)}^T={\mathbf{AB}}^T,$$

(3)

where the subscript $\left(k\right)$ stands for the first k columns of the matrix. We can define a biplot in the form of Equation (1) taking $\mathbf{A}={\mathbf{U}}_{\left(k\right)}{\Lambda }_{\left(k\right)}^{\gamma }$ and $\mathbf{B}=\mathbf{V}{\Lambda }_{\left(k\right)}^{(1-\gamma )},0\le \gamma \le 1$. Then, $\mathbf{\Theta }={\mathbf{AB}}^T$ minimizes the Frobenius norm

$${∥\mathbf{X}-\mathbf{\Theta }∥}_F^2=\sum _{i=1}^n{∥{\mathbf{x}}_i-\left({\mathbf{a}}_{i1}{\mathbf{b}}_1+\dots +{\mathbf{a}}_{ik}{\mathbf{b}}_k\right)∥}^2,$$

(4)

for any value of $\gamma$. Note that, for example, if $\gamma =1$, $\mathbf{A}$ contains the coordinates of individuals on the principal components and $\mathbf{B}$ is the projection of the initial coordinate axis onto them. There are algorithms other than SVD used to calculate the markers as alternated regressions [13] or NIPALS [35].

2.2. Logistic Biplot

Let $\mathbf{X}={\left({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\right)}^T$ be a binary matrix, with ${\mathbf{x}}_i\in {\{0,1\}}^p$, $i=1,\dots ,n$ and $x_{ij}\sim Ber\left(\pi \left({\theta }_{ij}\right)\right)$, where $\pi (·)$ is the inverse link function. In this paper, the logit link is used, $\pi \left({\theta }_{ij}\right)={\left\{1+exp(-{\theta }_{ij})\right\}}^{-1}$, which represents the expected probability that the character j is present in individual i, the log-odds $\pi \left({\theta }_{ij}\right)$ is ${\theta }_{ij}$ with ${\theta }_{ij}=log\left\{\pi \left({\theta }_{ij}\right)/(1-\pi \left({\theta }_{ij}\right))\right\}$, which corresponds to the natural parameter of the Bernoulli distribution expressed in an exponential family form. Using the probability distribution, we have $P(X_{ij}=x_{ij})=\pi {\left({\theta }_{ij}\right)}^{x_{ij}}{\left(1-\pi \left({\theta }_{ij}\right)\right)}^{1-x_{ij}}$, and the loss function is obtained as the negative log likelihood

$$\mathcal{L}\left(\mathbf{\Theta }\right)=-\sum _{i=1}^n\sum _{j=1}^p\left[x_{ij}log\left(\pi \left({\theta }_{ij}\right)\right)+(1-x_{ij})log(1-\pi \left({\theta }_{ij}\right))\right].$$

(5)

In this case, it is not appropriate to center the columns because the centered matrix will no longer be made up of elements equal to zero or one. Therefore, we extend the specification of the natural parameter space by introducing variable main effects or the column offset term $\mathit{\mu }$ for a model-based centering. The canonical parameter matrix $\mathbf{\Theta }={\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_n\right)}^T$ can be represented by a low-dimensional structure for some integer $k\le r$ that satisfies ${\mathit{\theta }}_i=\mathit{\mu }+{\sum }_{s=1}^k{a}_{is}{\mathbf{b}}_s$, $i=1,\dots ,n$, which expressed in matrix form is written as

$$\mathbf{\Theta }=logit\left(\mathbf{\Pi }\right)={\mathbf{1}}_n{\mathit{\mu }}^T+{\mathbf{AB}}^T,$$

(6)

where ${\mathbf{1}}_{\mathbf{n}}$ is an n-dimensional vector of ones; $\mathit{\mu }={\left({\mu }_1,\dots ,{\mu }_p\right)}^T$; $\mathbf{A}={\left({\mathbf{a}}_{\mathbf{1}},\dots ,{\mathbf{a}}_{\mathbf{n}}\right)}^T$ with ${\mathbf{a}}_{\mathbf{i}}\in {\mathbb{R}}^k,i=1,\dots ,n$; $\mathbf{B}=\left({\mathbf{b}}_{\mathbf{1}},\dots ,{\mathbf{b}}_{\mathbf{k}}\right)$ with ${\mathbf{b}}_{\mathbf{j}}\in {\mathbb{R}}^p,j=1,\dots ,k$; and $\mathbf{\Pi }=\pi \left(\mathbf{\Theta }\right)$ is the predicted matrix whose $ij$th element equals $\pi \left({\theta }_{ij}\right)$. Now, $\mathbf{\Theta }=logit(\mathbf{\Pi })$ is a biplot in the logit scale and log-odds ${\theta }_{ij}={\mu }_j+{\mathbf{a}}_i^T{\mathbf{b}}_j$.

In addition to the canonical parameter matrix $\mathbf{\Theta }$, the hyperparameter k must also be estimated to select the LB model. In general, in dimensionality reduction methods, the choice of k is problematic and presents a risk of overfitting [36]. We propose the use of a completely unsupervised procedure based on cross-validation for LB model selection to counteract such overfitting. Cross-validation is operationally simple and allows the selection of the number of dimensions at the point where a numerical criterion is minimized, which will indicate that choosing more dimensions would lead to the overfitting of the model.

Once the parameters have been estimated, the geometry of the biplot regression allows us to obtain the direction vector ${\mathit{g}}_j$ that projects a row marker of $\mathbf{A}$ to predict the values in column j [21,26]. For example, to find the coordinates for a fixed probability $\pi$ when $k=2$, we look for the point $(g_{j1},g_{j2})$ that predicts $\pi$ and that is on the biplot axis, i.e., on the line joining the points $(0,0)$ and $(b_{j1},b_{j2})$, which is calculated as

$$\begin{array}{cc}{g}_{j1}=\frac{(logit\left(\pi \right)-{\mu }_j)b_{j1}}{{\sum }_{s=1}^2{b}_{js}^2},\hfill & \hfill g_{j2}=\frac{(logit\left(\pi \right)-{\mu }_j)b_{j2}}{{\sum }_{s=1}^2{b}_{js}^2}.\end{array}$$

(7)

In the logistics biplot, as in the classic PCA biplot, all directions pass through the origin. In a PCA biplot for centered data, the origin represents the mean of each variable and the arrow shows the direction of increasing values. As the binary data cannot be centered (we keep the column offset term in the model), the origin does not represent any particular value of the probability. In the LB model, we represent the variables with arrows (segments) starting at the point that predicts 0.5 and ending at the point that predicts 0.75.

2.3. Parameter Estimation

Expressing the loss function given in (5) as $\mathcal{L}\left(\mathbf{\Theta }\right)={\sum }_{i=1}^n{\sum }_{j=1}^{p}f\left({\theta }_{ij}\right)$ and considering that the logit link function is used, $\pi \left({\theta }_{ij}\right)={(1-exp(-{\theta }_{ij}))}^{-1}$, the gradient is

$$\begin{array}{cc}\nabla f\left({\theta }_{ij}\right)\hfill & =-\left[x_{ij}\frac{1}{\pi \left({\theta }_{ij}\right)}\frac{\partial \pi \left({\theta }_{ij}\right)}{\partial {\theta }_{ij}}+(1-x_{ij})\frac{1}{1-\pi \left({\theta }_{ij}\right)}\frac{\partial (1-\pi \left({\theta }_{ij}\right))}{\partial {\theta }_{ij}}\right]\hfill \\ \hfill & =-\left[x_{ij}(1-\pi \left({\theta }_{ij}\right))-(1-x_{ij})\pi \left({\theta }_{ij}\right)\right]\hfill \\ \hfill & =\pi \left({\theta }_{ij}\right)-x_{ij}.\hfill \end{array}$$

(8)

The second derivative, ${\nabla }^{2}f\left({\theta }_{ij}\right)=\pi \left({\theta }_{ij}\right)(1-\pi \left({\theta }_{ij}\right))$ is a quadratic function satisfying $0\le {\nabla }^{2}f\left({\theta }_{ij}\right)\le 1/4$ as $x_{ij}$ has a Bernoulli distribution.

2.3.1. Estimation Using Nonlinear Conjugate Gradient Algorithms

Let $\nabla \mathcal{L}\left(\mathbf{\Theta }\right)$ be a matrix whose $ij$th element equals $\pi \left({\theta }_{ij}\right)-x_{ij}$, which will be expressed in matrix form as

$$\nabla \mathcal{L}\left(\mathbf{\Theta }\right)=\mathbf{\Pi }-\mathbf{X}.$$

(9)

As the ${\theta }_{ij}={\mu }_j+{\mathbf{a}}_i^T{\mathbf{b}}_j$, then the $\mathcal{L}\left(\mathbf{\Theta }\right)$ function involves the matrices $\mathbf{A}$ and $\mathbf{B}$ through $\pi \left({\theta }_{ij}\right)=\pi ({\mu }_j+{\mathbf{a}}_i^T{\mathbf{b}}_j)$. In this way, it is also possible to calculate the partial derivatives with respect to $\mu$, $\mathbf{A}$, and $\mathbf{B}$ (see Appendix A):

$$\frac{\partial \mathcal{L}}{\partial \mathit{\mu }}={\left(\mathbf{\Pi }-\mathbf{X}\right)}^T{\mathbf{1}}_n$$

(10)

$$\frac{\partial \mathcal{L}}{\partial \mathbf{A}}=\left(\mathbf{\Pi }-\mathbf{X}\right)\mathbf{B}$$

(11)

$$\frac{\partial \mathcal{L}}{\partial \mathbf{B}}={\left(\mathbf{\Pi }-\mathbf{X}\right)}^T\mathbf{A}$$

(12)

For the model to be identifiable and to avoid the elements of $\mathbf{B}$ becoming too small due to changes in the scale of $\mathbf{A}$ and not due to changes in the log-likelihood, it is usually required that ${\mathbf{A}}^T\mathbf{A}={\mathbf{I}}_k$ [9]. However, the maximum likelihood estimation of the model with this constraint tends to overfit the data [12]. In this paper, we use an approach that allows us to control the scale from the simultaneous updating of the parameters.

The initialization of the algorithm starts at point ${\mathbf{\Theta }}_0={\mathbf{1}}_n{\mathit{\mu }}_0^T+{\mathbf{A}}_0{\mathbf{B}}_0^T$, which can be configured by the user or by using a random initialization. For a random initialization, all elements of ${\mathbf{A}}_0={\left({\mathbf{a}}_{\mathbf{1}}^{\mathbf{0}},\dots ,{\mathbf{a}}_{\mathbf{n}}^{\mathbf{0}}\right)}^T$, ${\mathbf{B}}_0=\left({\mathbf{b}}_{\mathbf{1}}^{\mathbf{0}},\dots ,{\mathbf{b}}_{\mathbf{k}}^{\mathbf{0}}\right)$, and ${\mathit{\mu }}_0$ can be sampled from the uniform distribution. We choose the first search direction $d_l^T$ to be the steepest descent direction at the initial point ${\mathbf{\Theta }}_0$; then, a line search method is used to calculate the step-length parameter ${\alpha }_l$, and using a scalar ${\beta }_l$, we define the rule for updating the direction based on the gradient at every iteration and thus simultaneously update $\mathit{\mu }$, $\mathbf{A}$, and $\mathbf{B}$, thus updating the natural space of parameters $\mathbf{\Theta }$. The pseudocode is written formally in Algorithm 1.

Algorithm 1 CG Algorithm for fitting a LB model

Input $\mathbf{X}$

Output $\mathit{\mu },\mathbf{A},\mathbf{B}$

1: Initialize ${\mathit{\mu }}_0,{\mathbf{A}}_0,{\mathbf{B}}_0$

2: ${\mathbf{\Theta }}_0={\mathbf{1}}_n{\mathit{\mu }}_0^T+{\mathbf{A}}_0{\mathbf{B}}_0^T$

3: $\nabla {\mathcal{L}}_0=\nabla \mathcal{L}\left({\mathbf{\Theta }}_0\right)$

4: ${\mathit{d}}_0=-\nabla {\mathcal{L}}_0$

5: l = 0

6: repeat

7:  ${\mathbf{\Pi }}_{\mathit{l}}=\pi \left({\mathbf{\Theta }}_{\mathit{l}}\right)$

8:  ${\alpha }_{\mathit{l}}=arg{min}_{\alpha >0}\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}}+\alpha {\mathit{d}}_{\mathit{l}}\right)$

9:  ${\mathbf{A}}_{\mathit{l}+1}={\mathbf{A}}_{\mathit{l}}+{\alpha }_{\mathit{l}}\left({\mathbf{\Pi }}_{\mathit{l}}-\mathbf{X}\right){\mathbf{B}}_{\mathit{l}}$

10:  ${\mathbf{B}}_{\mathit{l}+1}={\mathbf{B}}_{\mathit{l}}+{\alpha }_{\mathit{l}}{\left({\mathbf{\Pi }}_{\mathit{l}}-\mathbf{X}\right)}^T{\mathbf{A}}_{\mathit{l}}$

11:  ${\mathit{\mu }}_{\mathit{l}+1}={\mathit{\mu }}_{\mathit{l}}+{\alpha }_{\mathit{l}}{\left({\mathbf{\Pi }}_{\mathit{l}}-\mathbf{X}\right)}^T{\mathbf{1}}_n$

12:  ${\mathbf{\Theta }}_{l+1}={\mathbf{1}}_n{\mathit{\mu }}_{l+1}^T+{\mathbf{A}}_{l+1}{\mathbf{B}}_{l+1}^T$

13:  $\nabla {\mathcal{L}}_{\mathit{l}+1}=\pi \left({\mathbf{\Theta }}_{\mathit{l}+1}\right)-\mathbf{X}$

14:  Compute ${\beta }_{\mathit{l}+1}$ according to one of the formulas given in (13)

15:  ${\mathit{d}}_{\mathit{l}+1}=-\nabla {\mathcal{L}}_{l+1}+{\beta }_{\mathit{l}+1}{\mathit{d}}_{\mathit{l}}$

16: until $\left(\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}}\right)-\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}+\mathit{1}}\right)\right)/\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}}\right)<ϵ$

This is a method that is considered to have low computational cost because each iteration only requires the evaluation of the gradient and the loss function, so it can become efficient in large data sets [37,38]. In this paper, we use four well-known formulas for the update direction: Fletcher–Reeves (FR), Polak–Ribiere–Polyak (PRP), Hestenes–Stiefel (HS), and Dai–Yuan (DY) [39,40,41,42], which can be written as

$$\begin{array}{c}\hfill {\beta }_{l+1}^{FR}=\frac{{∥\nabla {\mathcal{L}}_{l+1}∥}^2}{{∥\nabla {\mathcal{L}}_l∥}^2};{\beta }_{l+1}^{PRP}=\frac{\nabla {\mathcal{L}}_{l+1}^T{\Delta }_l}{{∥\nabla {\mathcal{L}}_l∥}^2};{\beta }_{l+1}^{HS}=\frac{\nabla {\mathcal{L}}_{l+1}^T{\Delta }_l}{d_l^T{\Delta }_l};{\beta }_{l+1}^{DY}=\frac{{∥\nabla {\mathcal{L}}_{l+1}∥}^2}{d_l^T{\Delta }_l},\end{array}$$

(13)

where ${\Delta }_l=\nabla {\mathcal{L}}_{l+1}-\nabla {\mathcal{L}}_l$, and $∥·∥$ is the Euclidean norm. These methods have had modifications that combine the formulas [43,44,45,46,47,48,49] and that could be adapted to fit a LB model.

To achieve convergence in the implementation of the CG algorithm, one often requires the step-length obtained with a line search to be exact or satisfy the strong Wolfe conditions,

$$\begin{array}{c}\mathcal{L}\left({\mathbf{\Theta }}_l\right)-\mathcal{L}\left({\mathbf{\Theta }}_l+{\alpha }_l{d}_l\right)\ge -c_1{\alpha }_l\nabla {\mathcal{L}}_l^T{d}_l,\\ \hspace{1em} ∥\nabla \mathcal{L}{\left({\mathbf{\Theta }}_l+{\alpha }_l{d}_l\right)}^T{d}_l∥\le -c_2\nabla {\mathcal{L}}_l^T{d}_l,\end{array}$$

(14)

with $0<c_1<c_2<1$. As the FR method has been shown to be globally convergent under strong Wolfe line searches with $c_2<1/2$ [50,51], this condition ensures that all $d_l$ directions are descending directions from $\mathcal{L}$.

2.3.2. Estimation Using Coordinate Descendent MM Algorithm

As the optimization problem is non-convex, we also use a MM algorithm to generate solutions that decrease with each iteration. The negative log-likelihood can be majorized by a quadratic function, and the majorized function can then be minimized iteratively.

According to Taylor’s theorem, and taking into account the fact that ${\nabla }^{2}f\left({\theta }_{ij}\right)\le 1/4$, the loss functions of a single estimated natural parameter given in (5) are quadratically approximated at ${\theta }_{ij}^{\left(l\right)}$ by

$$\begin{array}{cc}f\left({\theta }_{ij}\right)\hfill & =-\left[x_{ij}log\left(\pi \left({\theta }_{ij}\right)\right)+(1-x_{ij})log(1-\pi \left({\theta }_{ij}\right))\right]\hfill \\ \hfill & \approx f\left({\theta }_{ij}^{\left(l\right)}\right)+(\pi \left({\theta }_{ij}^{\left(l\right)}\right)-x_{ij})({\theta }_{ij}-{\theta }_{ij}^{\left(l\right)})+\frac{1}{2}\pi \left({\theta }_{ij}^{\left(l\right)}\right)(1-\pi \left({\theta }_{ij}^{\left(l\right)}\right)){({\theta }_{ij}-{\theta }_{ij}^{\left(l\right)})}^2\hfill \\ \hfill & \le f\left({\theta }_{ij}^{\left(l\right)}\right)+(\pi \left({\theta }_{ij}^{\left(l\right)}\right)-x_{ij})({\theta }_{ij}-{\theta }_{ij}^{\left(l\right)})+\frac{1}{8}{({\theta }_{ij}-{\theta }_{ij}^{\left(l\right)})}^2\hfill \\ \hfill & =f\left({\theta }_{ij}^{\left(l\right)}\right)+\frac{1}{8}{\left({\theta }_{ij}-{\theta }_{ij}^{\left(l\right)}+4(\pi \left({\theta }_{ij}^{\left(l\right)}\right)-x_{ij})\right)}^2-2{(\pi \left({\theta }_{ij}^{\left(l\right)}\right)-x_{ij})}^2\hfill \end{array}$$

(15)

$$\begin{array}{cc}& =\frac{1}{8}{\left({\theta }_{ij}-{\theta }_{ij}^{\left(l\right)}+4(\pi \left({\theta }_{ij}^{\left(l\right)}\right)-x_{ij})\right)}^2+C.\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hfill \end{array}$$

(16)

Consequently, the loss function for the whole canonical matrix of parameters is majorized by

$$\mathcal{L}\left(\mathbf{\Theta }\right)\le \frac{1}{8}\sum _{i=1}^n\sum _{j=1}^p{\left({\theta }_{ij}-z_{ij}^{\left(l\right)}\right)}^2+C,$$

(17)

where C is a constant that does not depend on $\mathbf{\Theta }$, and $z_{ij}^{\left(l\right)}={\theta }_{ij}^{\left(l\right)}+4\left(x_{ij}-\pi \left({\theta }_{ij}^{\left(l\right)}\right)\right)$.

Let ${\mathbf{Z}}_l$ be the matrix whose $ij$th element equals $z_{ij}^{\left(l\right)}$. According to [52], the iteratively weighted least squares algorithm can be used as an upper bound for a quadratic function, which allows for the minimization of the majorization function as

$$\mathcal{L}\left(\mathbf{\Theta }\right)\le \frac{1}{8}{∥\mathbf{\Theta }-{\mathbf{Z}}_l∥}_F^2+C,$$

(18)

and thus the majorized function to be minimized is written as

$$\underset{\mathbf{\mu },\mathbf{A},\mathbf{B}}{min}{∥{\mathbf{1}}_n{\mathbf{\mu }}^T+{\mathbf{AB}}^T-{\mathbf{Z}}_l∥}_F^2.$$

(19)

In this case, the parameters are estimated sequentially. The algorithm is based on the coordinate descent optimization of the majorized function. When fixing ${\mathbf{AB}}^T$ in Equation (19), $\mathbf{\mu }$ can be estimated by the vector of mean scores of each column of matrix ${\mathbf{Z}}_l$, $\mathbf{\mu }=\frac{1}{n}{\mathbf{Z}}_l{\mathbf{1}}_n$. In this way, the optimization problem becomes ${min}_{\mathbf{A},\mathbf{B}}{∥{\mathbf{AB}}^T-\mathbf{P}{\mathbf{Z}}_l∥}_F^2$ where $\mathbf{P}=\mathbf{I}-\frac{1}{n}{\mathbf{1}}_n{\mathbf{1}}_n^T$. To initialize ${\mathbf{\Theta }}_0={\mathbf{1}}_n{\mathbf{\mu }}_0^T+{\mathbf{A}}_0{\mathbf{B}}_0^T$, we use ${\mathbf{A}}_0={\left({\mathbf{a}}_{\mathbf{1}}^{\mathbf{0}},\dots ,{\mathbf{a}}_{\mathbf{n}}^{\mathbf{0}}\right)}^T$; ${\mathbf{B}}_0=\left({\mathbf{b}}_{\mathbf{1}}^{\mathbf{0}},\dots ,{\mathbf{b}}_{\mathbf{k}}^{\mathbf{0}}\right)$. ${\mathbf{\mu }}_0$ can be sampled from the uniform distribution. Algorithm 2 presents the pseudocode of this process.

Algorithm 2 Coordinate descendent MM algorithm for fitting a LB model

Input $\mathbf{X}$

Output $\mathbf{\mu },\mathbf{A},\mathbf{B}$

1: Initialize ${\mathbf{\mu }}_0,{\mathbf{A}}_0,{\mathbf{B}}_0$

2: ${\mathbf{\Theta }}_0={\mathbf{1}}_n{\mathbf{\mu }}_0^T+{\mathbf{A}}_0{\mathbf{B}}_0^T$

3: l = 0

4: repeat

5:   ${\mathbf{\Pi }}_{\mathit{l}}=\pi \left({\mathbf{\Theta }}_{\mathit{l}}\right)$

6:   ${\mathbf{Z}}_l={\mathbf{\Theta }}_l+4\left(\mathbf{X}-{\mathbf{\Pi }}_{\mathit{l}}\right)$

7:   ${\mathbf{\mu }}_{(l+1)}=\frac{1}{n}{\mathbf{Z}}_l{\mathbf{1}}_n$

8:   ${\mathbf{Z}}_{l+1}^c=\mathbf{P}{\mathbf{Z}}_l$, with $\mathbf{P}=\mathbf{I}-\frac{1}{n}{\mathbf{1}}_n{\mathbf{1}}_n^T$

9:   ${\mathbf{Z}}_{l+1}^c=\mathbf{U}\Lambda {\mathbf{V}}^T$

10:    ${\mathbf{A}}_{l+1}=\mathbf{U}\Lambda$

11:  ${\mathbf{B}}_{l+1}=\mathbf{V}$

12:  ${\mathbf{\Theta }}_{l+1}={\mathbf{1}}_n{\mathbf{\mu }}_{l+1}^T+{\mathbf{A}}_{l+1}{\mathbf{B}}_{l+1}^T$

13: until $\left(\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}}\right)-\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}+\mathit{1}}\right)\right)/\mathcal{L}\left({\mathbf{\Theta }}_{\mathit{l}}\right)<ϵ$

2.4. Simulated and Real Data

2.4.1. Real Data

We used data from the Genomic Determinants of Sensitivity in Cancer 1000 (GDSC1000) [5]. The database contains 926 tumor cell lines with comprehensive measurements of point mutation, CNA, methylation, and gene expression. For the purpose of illustrating the methods, the methylation data were used, and to facilitate the interpretation of the results, three types of cancer were included: breast invasive carcinoma (BRCA), lung adenocarcinoma (LUAD), and skin cutaneous melanoma (SKCM).

2.4.2. Simulation Process

The data sets were simulated from a latent variables model with different levels of sparsity and with a low-dimensional structure according to the model presented in Equation (6). To generate the binary matrix $\mathbf{X}$, we used the procedure presented in Algorithm 3.

Algorithm 3 Algorithm to simulate a binary data matrix

Input $n,p,k,D$.

Output $\mathbf{X},\mathbf{\Theta },\mathbf{\mu },\mathbf{A},\mathbf{B}$

1: $\mathbf{B}= \left({\mathbf{b}}_{\mathbf{1}},\dots ,{\mathbf{b}}_{\mathbf{k}}\right)$ with ${\mathbf{b}}_{\mathbf{j}}\sim \mathcal{N}(0,1),j=1,\dots ,k$.

2: procedure Gram–Schmidt algorithm

3:  ${\mathbf{B}}^T\mathbf{B}={\mathbf{I}}_k$

4: end procedure

5: $\mathbf{\mu }={\left({\mu }_1,\dots ,{\mu }_p\right)}^T$ with ${\mu }_i=ln\left({\displaystyle \frac{D}{1-D}}\right)$

6: $\mathbf{A}\sim \mathcal{N}(\mathbf{0},{\mathbf{I}}_k)$

7: $\mathbf{\Theta }={\mathbf{1}}_n{\mathbf{\mu }}^T+{\mathbf{AB}}^T$

8: $\mathbf{X}={\left({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\right)}^T$, with $x_{ij}\sim Ber\left(\pi \left({\theta }_{ij}\right)\right),\pi \left({\theta }_{ij}\right)={\left\{1+exp(-{\theta }_{ij})\right\}}^{-1}$.

The offset term $\mathbf{\mu }$ was used to control the sparsity, while the log-odds $\mathbf{\Theta }$ was defined to have a low-dimensional structure.

2.5. Model Performance Assessment

To evaluate the performance of the CG algorithm and the MM algorithm, we used the training error, which is defined as the average misclassification rate when using the low-dimensional structure generated by the LB model with the training set. Each algorithm provided an estimate of $\mathbf{\mu },\mathbf{A}$, and $\mathbf{B}$. We used the predicted probability matrix $\mathbf{\Pi }=\pi \left({\mathbf{1}}_{\mathbf{n}}{\mathbf{\mu }}^{\mathbf{T}}+\mathbf{A}{\mathbf{B}}^{\mathbf{T}}\right)$ to select p thresholds, $0<{\delta }_j<1, j=1,\dots ,p$—one for each column of $\mathbf{X}$—and then perform the binary classification. As the data matrices could be imbalanced, we used the training error defined as Balanced Accuracy (BACC), which is most appropriate in these cases [53,54]

$$BACC=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right),$$

(20)

where $TP$ is the number of true positives, $TN$ is the number of true negatives, $FP$ is the number of false positives, and $FN$ is the number of false negatives.

To decide the classification rule from the predicted value, a threshold must be selected for each variable. This threshold value can be selected by minimizing the BACC rate for each variable in the training set and then applying the rule to a test set.

As the training error can be an optimistic measure of the misclassification rate, we also used a cross-validation procedure that allowed for the testing of the models; thus, we used a test data set that was independent of the training data set. As in the supervised models, cross-validation was used to keep track of the overfitting of the model. In our case, the LB model is a non-supervised method to reduce the dimensionality, and cross-validation helps to find the number of dimensions to maintain [36,55,56,57]. The procedure allowed us to estimate the value of the hyperparameter k to avoid overfitting and evaluate the capacity of the different estimation algorithms to identify the low-dimensional space.

As in PCA, we expect that a few dimensions would reproduce the original data (binary) as well as possible; then, we excluded some observed values, fit the model without those values, and calculated the expected probabilities for the missing data. Using the rule of classifying a missing value as being present if the fitted probability is higher than a threshold ${\delta }_j$, we imputed the selected missing data and calculated the classification. We performed this procedure M times for different values of k, and select the value that minimizes the generalization error.

More formally, we used the exclusion pattern proposed by Wold [55], which consists of a selected sequence of elements $x_{ij}$ being left out using a diagonal pattern and treated as missing values; in this way, we avoided the problems of excluding a complete row or column as described in [36,57]. Let $\mathbf{W}$ be a binary matrix, where $w_{ij}=0$ if the value is excluded (belongs to the validation set) and $w_{ij}=1$ is not (belongs to the training set). The minimization problem thus became

$$\begin{array}{cc}\underset{\mathbf{\mu },\mathbf{A},\mathbf{B}}{min}\hfill & -log\left(p\right(\mathbf{X};\mathbf{\Theta },\mathbf{W}\left)\right)\hfill \\ \hfill & =-log\left({\displaystyle \prod _{i=1}^n\prod _{j=1}^p}{\left[p(x_{ij};{\theta }_{ij})\right]}^{w_{ij}}\right)\hfill \\ \hfill & =-{\displaystyle \sum _{i=1}^n\sum _{j=1}^p}{w}_{ij}\left[x_{ij}log\left(\pi \left({\theta }_{ij}\right)\right)+(1-x_{ij})log(1-\pi \left({\theta }_{ij}\right))\right].\hfill \end{array}$$

(21)

The loss function now depends only on the training set. Solving the problem with one of the proposed algorithms, we created a matrix of expected probabilities $\mathbf{\Pi }=\pi (\mathbf{\Theta })$. Then, ${\widehat{x}}_{ij}=1$ when $\pi (\mathbf{\Theta })>{\delta }_j$, and ${\widehat{x}}_{ij}=0$ otherwise, where $0<{\delta }_j<1$ is the threshold for minimizing the BACC rate for variable j in the training set. We used the elements of $\widehat{\mathbf{X}}$ when $w_{ij}=0$ to calculate the BACC. This procedure was performed M times, and the generalization error given by the cross-validation was calculated using the mean of the BACC; we considered $M=7$ folds as in [57].

We also measured the models’ ability based on each algorithm to recover the log-odds $\mathbf{\Theta }$; for this, we used the relative mean squared error (RMSE), defined as

$$RMSE(\mathbf{\Theta })=\frac{{∥\mathbf{\Theta }-\widehat{\mathbf{\Theta }}∥}_F^2}{{∥\mathbf{\Theta }∥}_F^2}.$$

(22)

where $\mathbf{\Theta }$ is the true parameter and $\widehat{\mathbf{\Theta }}$ is estimated by the LB model using one of the proposed algorithms.

3. Results

3.1. Monte Carlo Study

Binary matrices were simulated with $n=100, 300, 500$; $p=50, 100$; $k=3$; and $D=0.5, 0.3, 0.2, 0.1$, where the parameter D represents the proportion of ones in matrix $\mathbf{X}$. The different sparsity levels were simulated to check if the performance with the algorithms to find the low-dimensional structure was affected for this reason. The combinations of n, p, k, and D generated the different scenarios; in each scenario, R matrices were simulated independently, and the measures were calculated to evaluate the performance of the algorithms. Finally, the mean of the cross-validation error (cv error) was calculated, as well as the mean of the training error (BACC) and the mean of the relative mean squared error (RMSE) with their respective standard errors. We used a value of $R=30$ that generated standard errors less than $1\%$ in the estimation of the BACC and cv error.

Figure 1a presents the cross-validation error of the algorithms based on the conjugate gradient and the MM algorithm when the matrix $\mathbf{X}$ is balanced. From the cv error, we can see that all models began to overfit when $k>3$, so the five estimation methods identified the three underlying dimensions for all balanced scenarios that were simulated. Figure 1b shows the Balanced Accuracy (BACC); it is highlighted that the slope decelerated when the three underlying dimensions were reached in the matrix, and thus an elbow criterion could be used as a signal of the number of dimensions to choose. Finally, Figure 1c shows the estimation of the relative mean squared error (RMSE) for the matrix $\mathbf{\Theta }$; we see that the algorithms based on the conjugate gradient and the MM algorithm showed similar results when the number of dimensions was less than or equal to 3. Whereas, when the model had more than the three predefined dimensions in the simulation, the algorithms based on the conjugate gradient presented a lower RMSE than the MM algorithm, although, for a fixed value of p, these gaps closed as n increased.

Figure 2a–c show the cross-validation errors when the data are imbalanced with $D=0.3, 0.2$, and $0.1$, respectively. In all the scenarios studied, it is observed that the error was minimized when the number of underlying dimensions in the space of the variables was reached, so our method identified that a value of $k=3$ in the LB model was appropriate to avoid overfitting. As this occurred with all imbalanced data sets, then we see that the level of dispersion does not affect the performance of the algorithms in terms of correctly finding the low-dimensional space.

The training error for imbalanced data sets with different levels of sparsity is shown in Figure 3a–c. In all the studied scenarios, it is observed that the percentage of loss in the training error stabilized from the third dimension. In this way, the different algorithms allowed the low-dimensional space to be appropriately selected using the elbow method.

The RMSE of the estimation of the log-odds $\mathbf{\Theta }$ for the different levels of sparsity is shown in Figure 4a–c; we can see that the algorithms presented similar performances. In the scenarios of $n=100$ and $p=50$, it is observed that the RMSE increased notably when the number of dimensions was greater than 3, so there were some important gaps between the two approaches, although these differences decreased as the value of p or the value of n increased.

On the other hand, the computational performance of the algorithms was measured on a computer with an Intel Core i7-3517U processor with 6 GB of RAM.

Table 1. Running time in seconds to estimate the LB model with $k=3$ and $ϵ={10}^{-4}$.

n	p	D	DY	FR	HS	PR	MM
100	50	0.1	29.9	30.1	29.9	29.9	166.1
300	50	0.1	84.6	85.4	85.0	85.3	276.3
500	50	0.1	141.2	141.5	141.9	141.1	356.3
100	100	0.1	58.2	57.8	57.9	58.6	115.1
300	100	0.1	168.6	167.4	168.9	169.4	155.3
500	100	0.1	302.4	279.6	330.8	322.3	215.5
100	50	0.2	30.1	30.1	30.3	30.1	60.8
300	50	0.2	102.8	103.1	102.5	100.8	133.3
500	50	0.2	141.7	159.3	142.4	141.9	140.1
100	100	0.2	62.4	58.4	63.6	58.6	35.7
300	100	0.2	169.8	169.0	168.4	168.9	67.3
500	100	0.2	283.0	288.0	281.9	283.1	99.5
100	50	0.3	30.1	30.6	30.2	30.4	36.6
300	50	0.3	86.2	86.6	86.2	86.2	60.1
500	50	0.3	143.1	143.2	143.0	143.9	99.3
100	100	0.3	58.7	59.0	58.8	58.8	26.8
300	100	0.3	170.5	170.5	170.3	169.7	50.1
500	100	0.3	284.2	284.5	284.4	284.6	77.3
100	50	0.5	30.2	31.1	32.6	30.9	27.6
300	50	0.5	87.1	98.7	87.5	86.9	55.8
500	50	0.5	145.6	144.6	145.4	144.8	103.0
100	100	0.5	58.7	58.9	59.2	59.2	18.8
300	100	0.5	170.8	171.2	171.0	171.1	30.2
500	100	0.5	286.4	327.9	290.6	341.6	56.4

shows the the running time in seconds with 100 replications for $k=3$ and a stopping criterion of $ϵ={10}^{-4}$. We see that the performances of the different algorithms were competitive, and they converged relatively quickly when the maximum of the absolute changes of the estimated parameters in two subsequent iterations was less than ${10}^{-4}$. In general, it is observed that the CPU times of the CG algorithms were similar and presented a better performance than the MM algorithm when $p=50$ and the sparsity level began to be high ($D\le 0.2$), or when $n=100$, $p=100$, and $D=0.1$. In the other cases, the MM algorithm performed better, especially when n and p increased, resulting in up to six times faster performance than CG algorithms in balanced scenarios when $n=500$ and $p=100$.

3.2. Application

To apply the proposed methodology, we used data from the Genomic Determinants of Sensitivity in Cancer 1000 (GDSC1000) [5]. The database contains 926 tumor cell lines with comprehensive measurements of point mutation, CNA, methylation, and gene expression. To illustrate the methods, the methylation data were used, and to facilitate the interpretation of the results, three types of cancer were included: breast invasive carcinoma (BRCA), lung adenocarcinoma (LUAD), and skin cutaneous melanoma (SKCM).

We performed preprocessing to sort the data sets and separate the methylation information into one data set. After preprocessing, the methylation dataset has 160 rows and 38 variables, each variable is a CpG island located in the gene promoter area. In this case, code 1 indicates a high level of methylation, and 0 indicates a low level; approximately $27\%$ of the binary data matrix are ones.

Figure 5 shows the cross-validation error and training error using the conjugate gradient algorithms and the coordinate descendent MM algorithm. If $k=0$, the model (6) only considered the term $\mathbf{\mu }$, meaning that $\mathbf{\Theta }={\mathbf{1}}_n{\mathbf{\mu }}^T$, where ${\mu }_j$ is the proportion of ones in column j and was used as a reference to observe the performance of the algorithms when more dimensions were included by incorporating the row and column markers, $\mathbf{\Theta }={\mathbf{1}}_n{\mathbf{\mu }}^T+{\mathbf{AB}}^T$.

The cross-validation error was minimized in three dimensions for the four formulas (FR, HS, PR, and DY) based on the CG algorithm, so $k=3$ was the appropriate value to avoid overfitting when using these methods of estimation. When using the MM algorithm, it was found that the LB model generated overfitting for $k>2$, so using two dimensions in the LB model is suitable when using this estimation method.

An advantage of using a biplot approach is that it allows for a simultaneous representation of rows and columns, which are plotted with points and directed vectors, respectively. Figure 6 shows the biplot obtained for the methylation data using the Fletcher–Reeves conjugate gradient algorithm; the vectors of the variables are represented by arrows (segments) that start at the point that predicts $0.5$ and end at the point that predicts $0.75$. Therefore, short vectors indicate a rapid increase in probability and the orthogonal projection of the row markers on the vector approximates the probability of finding high levels of methylation in the cell line.

The position of the segment, which corresponds to the point that predicts a probability of 0.5, can start any side around the origin. For example, in Figure 6, the variable DUSP22 points to the origin, when doing the orthogonal projection of the points in the direction of the vector, most of them will be projected after the reference point where the segment starts; this means that almost all cell lines of the three groups have high fitted probabilities of having high levels of methylation in that variable.

The cell lines are separated into three clearly identified clusters. In the BRCA type of cancer, variables such as NAPRT1, THY1, or ADCY4 are directed towards the positive part of dimension 1 and therefore have a greater probability of presenting high levels of methylation. The LUAD cell lines are located in the negative part of dimension 2, so these have a high propensity to present high levels of methylation in variables such as HIST1H2BH, ZNF382, and XKR6. Finally, the cell lines for the SKCM cancer type are located in the negative part of dimension 1 and have a greater probability of presenting high levels of methylation in variables such as LOC100130522, CHRFAM7A, or DHRS4L2.

Table 2. Sensitivity and specificity for each variable when fitting the LB model with $k=3$ and a CG-FR algorithm.

Variable	Sensitivity	Specificity	Global
GSTM1	97.1	94.8	96.2
C1orf70	100.0	100.0	100.0
DNM3	100.0	94.9	96.2
COL9A2	100.0	100.0	100.0
VAR5	100.0	94.3	95.6
VAR6	100.0	88.6	91.2
THY1	100.0	95.8	96.9
VAR8	100.0	93.6	95.0
DNAH10	100.0	100.0	100.0
VAR10	100.0	90.8	92.5
DHRS4L2	100.0	88.8	91.2
ADCY4	100.0	98.3	98.8
CHRFAM7A	100.0	92.4	94.4
VAR14	100.0	99.1	99.4
FAM174B	100.0	94.8	96.2
VAR16	100.0	99.1	99.4
VAR17	100.0	86.3	87.5
ARL17A	100.0	98.6	99.4
HOXB8	100.0	98.2	98.8
LOC100130522	100.0	89.3	91.9
ZNF714	100.0	100.0	100.0
ZNF382	97.9	94.6	95.6
VAR23	100.0	96.5	97.5
FRZB	100.0	99.1	99.4
VAR25	100.0	94.7	95.0
TBX1	100.0	98.3	98.8
LOC391322	100.0	80.9	81.2
GSTT1	72.0	87.1	80.0
VAR29	100.0	96.6	97.5
FILIP1L	97.1	93.6	94.4
VAR31	100.0	97.5	98.1
HIST1H2BH	100.0	89.7	92.5
DUSP22	100.0	98.4	99.4
VAR34	95.0	97.5	96.9
XKR6	100.0	93.2	95.0
NAPRT1	96.9	87.5	89.4
VAR37	95.8	95.6	95.6
VAR38	97.7	98.3	98.1

shows the rate of correct classifications for each variable using the measures of sensitivity and specificity; these measures allowed us to determine if the model exhibited a good classification for the two types of data in each variable. Sensitivity measured the true positives rate, specificity measured the true negatives rate, and the global measure corresponded to the total rate of correct classifications for each variable.

In general, the model with three dimensions and using the CG algorithm with the FR formula generated high values for sensitivity; only the GSTT1 gene presented a relatively low sensitivity, with 72% true positives. Regarding specificity, the LOC391322 gene obtained the lowest true negatives rate, with 80.9%. Thus, the results of the model are satisfactory.

4. Conclusions and Discussion

The Logistic Biplot (LB) model is a dimensionality reduction technique that generalizes the PCA to deal with binary variables and has the advantage of simultaneously representing individuals and variables (biplot).

In this paper, we propose and develop a methodology to estimate the parameters of the LB model using nonlinear conjugate gradient algorithms or the descending coordinate MM algorithm. For the selection of the LB model, we have incorporated a cross-validation procedure that allows the choice of the number of dimensions of the model in order to counteract overfitting.

As a complement to the proposed methods and to give practical support, a package has been written in the R language called BiplotML package [33], which is available from CRAN; this is a valuable tool that enables the application of the proposed algorithms to data analysis in every scientific field. Our contribution is important because we provide alternatives to solve some problems encountered in the LB model in the presence of sparsity or a big data matrix [23,26,27,28]. Additionally, a procedure is presented that allows the choice of the number of dimensions, which until now had not been investigated for an LB model.

The proposed algorithms are iterative and have the property that the loss function decreases with each iteration. To study the properties of the proposed algorithms to fit a LB model, low rank data sets with $k=3$ and different levels of sparsity were generated for $n=100, 300, 500$ rows and $p=50, 100$ columns. The accuracy of the algorithms was measured using the training error, generalization error (cv-error), and mean square error (RMSE) of the log-odds. According to the Monte Carlo study, we established that the cross-validation criterion is successful in the estimation of the hyperparameter of the number of dimensions. This allows the model to be specified and thus avoid overfitting; in this way, we obtain the best performance of the proposed algorithms in terms of recovering the underlying low rank structure.

The comparison of the running times showed that the algorithms converge quickly. The CG algorithm is more efficient when the matrices are sparse and not very large, while the performance of the MM algorithm is better when the number of rows and columns tends to increase; thus, it is preferable for large matrices.

Finally, we used real data on gene expression methylation to show our approach. The LB model allowed us to carry out a simultaneous projection between rows and columns, where a grouping of three classes was observed, formed by the cell lines of the three types of cancer analyzed. Furthermore, the vectors that represented the variables allowed us to identify those cell lines that were more likely to achieve high levels of methylation in the different genes.