The combination of biomarkers (so-called biomarker signature) allows us to represent the information contained in biological samples and fluids, supporting clinical decisions [1
]. Numerous studies demonstrated the clinical usefulness of diagnostic (disease detection) and prognostic (disease outcome) gene-expression signatures derived from microarray analysis [2
]. For instance, MammaPrint is a 70 gene-expression prognostic signature for powerful disease outcome prediction in breast cancer [4
]. The diagnostic miR-Test shows promising results for lung cancer early detection [5
However, reliable clinical signatures are restricted by dataset availability, which often reduces their statistical power [3
]. Artificially increasing the number of samples by combining different large cohorts using dataset merging (meta-analysis) is a beneficial solution enabling numerous insights into biological systems [7
], but methods for biomarker signature identification are currently limited. For instance, the R packages virtualArray [11
] and inSilicoMerging [12
] allow virtual array merging but are no longer available and are removed from current Bioconductor releases [13
]. On the other hand, database tools such as SurvExpress [14
] and SurvMicro [3
] allow for the assessment of a prognostic signature in cancer. Similarly, the miRpower tool provides survival analysis for miRNA biomarkers using expression data from 2178 breast cancer patients [15
] and GOBO based on 1881 breast cancer dataset [16
], whereas the Kaplan-Meier Plotter enables outcome analysis for ovarian cancer based on 1287 patients [17
]. However, these tools focus on specific diseases and signature types. More importantly, they allow only online analysis, requiring a gene list as input, but not the calculation of signatures from in-house data. These characteristics limit them as stand-alone tools, suggesting new bioinformatics approaches.
Machine learning (ML) approaches have been demonstrated to be useful in medicine. For example, studies report that ML could be used in cancer diagnosis [18
] and prognosis [19
] as well as prediction of optimal cancer therapies [20
]. It can also improve the prediction of heart failure readmissions [21
Regularized Generalized Linear Models such as L1/L2 regularized and Elastic net regression address overfitting and aim to balance between accuracy and simplicity of a model [22
]. The Least Absolute Shrinkage and Selection Operator (LASSO) uses L1 regularization, whereas Elastic net implements a mixture of L1 and L2 regularization. Applying these regularization techniques to fit a Generalized Linear Model is widely used for feature selection and is extremely effective when dealing with high dimensional data, which contains a large set of features. The LASSO model allows the shrinkage of the coefficients of the less contributive variables to be exactly zero (the penalty term L1-norm) [22
]. Thereby, the tuning parameter lambda controls the strength of the penalization (regularization). The cross-validation calculates the lambda.min value, which reflects the model with the lowest prediction error, whereas the lambda.1se value represents a simpler model but within one standard error of the optimal model. However, the LASSO regression tends to over-regularization and has limited strength in highly correlated data.
The Elastic net balances between LASSO (L1-norm) and ridge penalties (L2-norm) shrinking some coefficients close to zero (like ridge) and some exactly to zero (similar to LASSO) [23
]. This model is powerful in datasets with e.g., correlations between variables. For this, the hyper-parameter alpha controls the mixing between the two penalty techniques (alpha = 0 for ridge; alpha = 1 for LASSO) and can be set manually between 0 and 1 to receive a model with the desired size, whereas the parameter lambda fine-tunes the amount of shrinkage [23
]. Therefore, the Elastic net is a powerful method for feature selection and can operate with continuous as well as categorical features.
Several statistical methods have been developed for survival data analysis [24
]. The Cox Proportional Hazard model is the most popular multivariate approach to investigate survival time in medical research [24
]. It describes the relation between event incidence (hazard function, survival probability) and covariates [24
We previously introduced a sample merging approach that is compatible with current Bioconductor releases [27
]. It allows the use of datasets from databases such as Gene Expression Omnibus (GEO), The Cancer Genome Atlas (TCGA), and own experimental data [27
], greatly enhancing the number of available datasets for analysis. Starting from this, we developed a protocol for the systematical calculation of diagnostic and prognostic gene signatures that combines (i) meta-analysis (multiple dataset integration) with (ii) functional gene expression analysis and (iii) ML approaches. Our aim was to develop a general framework for functional analysis and signature calculation with high predictive performance that is not restricted to specific disease types and can therefore be used by a broad community.
Our intention was to develop a general and easy to use toolbox that identifies reliable diagnostic and prognostic signatures including the important steps of data augmentation and validation, especially for users with limited bioinformatics resources. It is therefore a step-by-step protocol rather than an improved algorithm or ML method approach.
The tool applies a comparison between the two ML models LASSO and Elastic net, which aim to balance between accuracy and simplicity of a model. LASSO and Elastic net regularization are well-established methods for gene expression analysis, allowing to construct predictive models from datasets with non-linear and large dimensional variable numbers [21
]. Especially for generalization of data with additive variable and outcome dimensions or a low number of training datasets they generate predictive results similar to complex ML algorithms [19
]. Complex ML approaches such as support vector machines, neural networks, random forest, and gradient boosting algorithms allow unbiased predictive models using complex variable selection and huge datasets but tend to overfitting in the identification of large biomarker combinations [1
]. However, the combinations of biomarkers show better discriminatory power for clinical decision support rather than a single biomarker [1
The use of ML implies the need for a substantial amount of data in order to train the model, in which the integration of different datasets might be required. However, gene expression analysis often suffers from selection bias, poor sample quality, and poor sample size estimation, influencing the statistical power and validity of downstream analysis [1
]. Combing different gene expression datasets using meta-analysis has been shown to increase statistical power and overcome selection biases including the identification of diagnostic and prognostic biomarkers [7
]. However, differentially gene expression selection using meta-analysis is mostly based on univariate p-value statistics which introduces the problem to identify sets of genes with non-redundant information and to find the correct number of genes that describe the data [8
]. This limits application for diagnostic and prognostic signatures that integrate several feature selections and covariates such as patient characteristics (e.g., survival) and histology [8
]. We overcome this by implementing a meta-analysis for the integration of multiple gene expression datasets into a merging array and then applied ML methods to identify biomarker signatures from datasets with non-linear and large dimensional variable numbers.
Several studies calculate signatures using ML approaches, but often fail during independent validation stages [36
]. To overcome overfitting and reduce model instability, we identified a classifier in the training dataset and applied a comprehensive evaluation using different validation strategies—in particular, a split sample, internal validation (cross-validation) and testing in independent datasets. Moreover, we applied a multiple-testing correction using the Benjamini and Hochberg method and set a stringent q-value of 0.05. We recommend using a stringent q
-value (can be set by the user) to reduce the false positives and find real biologically deregulated genes but also considering sample size and power estimation approaches based on statistical and clinical significance [1
]. This strengthens the robustness for the biomarker signature identification capability and validity for clinical usefulness.
In our example, the identified gene signatures from two different ML models show a high diagnostic power and might be promising for the clinic to classify between lung cancer and non-cancer samples. The confusion matrix for the LASSO and Elastic net regression models are similar. Comparing the calculated signatures shows a common set of 12 transcripts (12 unique gene symbols), and similar accuracy and predictive performance. However, this is of course not always the case. For example, studies in breast cancer reported two independent prognostic signatures identified with similar approaches showing only few common genes, which were experimentally validated [42
]. This illustrates that different mathematical models should be applied to find the most reliable signature rather than using only one method. Hence, using several methods reduces false positive results even for challenging datasets and avoids misclassification in experimental and clinical testing. This strengthens the validity and clinical usefulness of signatures extracted from large gene expression datasets.
The common gene set contains known cancer markers. For instance, TMEM106B has been shown to be a valuable marker of lung cancer metastasis [43
], whereas COL10A1 [44
] plays a diagnostic role of circulating extracellular matrix-related proteins. However, LGR4 [45
] is known as a diagnostic marker in prostate cancer. This highlights that our analysis approach allows the identification of reliable diagnostic signatures. The next step is then to validate and iteratively refine the marker signature derived from our tool in prospective clinical studies to find an optimal biomarker signature, with the help of more complex ML models.
The significance and novelty of the toolbox lies in its functionality as an „all-in-one tool”: it offers an analysis path combining meta-analysis with functional gene expression analysis and robust diagnostic and prognostic signature calculation. The code is implemented in an R package. The four main functions—sigidentDEG, sigidentEnrichment, sigidentDiagnostic, and sigidentPrognostic—are wrapper functions around all included smaller functions to execute the analysis steps. However, these can also be run separately, depending on the interests of the users.
The toolbox benefits from its generic design and modularity. We designed it for Affymetrix as a widely used microarray profiling platform [46
] and illustrate the generality of the approach using lung cancer gene expression datasets (tumor/healthy) downloaded from the GEO database. The generic design of the tool allows the analysis of different types of gene expression signatures, e.g., mRNA, lncRNA, and miRNA. Furthermore, it supports analysis in front of the high biological complexity of tumors, for instance analysis of tumor subtypes and heterogeneity.
We demonstrated the method’s power to be applied to datasets containing a large number of gene probes using the Affymetrix HG-U133 Plus 2.0 platform. However, the merging algorithm is not restricted to this platform, allowing the potential integration of other popular microarray profiling platforms such as HG-U133A, HG-U133B, and HG-U133A 2.0. Moreover, the modularity of the framework allows the future incorporation of additional platforms, such as Illumina, but also other high-throughput data such as genomic, proteomic, metabolomic, and radiomic data. For instance, the Elastic net model shows applicability to genome-scale data such as the identification of genomic markers of drug sensitivity [8
]. Indeed, the implementation of this complex data requires programming skills and is therefore recommended only for experienced users. Such a broad applicability is in principle possible but was not the intention of the current version of the framework and should be the focus of future work. Further efforts should also focus on the integration of the toolbox into a web application to provide its functionality to users without R programming skills.
Existing tools such as SurvMicro [3
] and SurvExpress [14
] allow for the online validation of prognostic signatures, but are restricted to datasets from TCGA and limited to cancer. Our toolbox has the advantage to be disease independent and allows the integration of data from TCGA and GEO, but also from in-house experiments.
The framework from Hughey et al. 2015 identifies a diagnostic signature combining meta-analysis with an Elastic net regression [8
]. This approach is similar to our method, but our tool calculates prognostic signatures as a further relevant biomarker signature for clinical application. Additionally, the regularization methods LASSO and Elastic net can be applied for the aim of feature selection to identify variables correlated to the desired response variable. The toolbox also integrates an automated method to identify DEGs, including a summary table with gene annotations and functional enrichment analysis. In this way, our method can also be used to perform a functional DEG analysis from merged datasets without the calculation of signatures. In conclusion, the user-friendly R package, the all-in-one functionality, and modularity make the framework useful to a broad community.
4. Materials and Methods
illustrates the workflow of our toolbox. It has been developed and tested on R version 3.6.1 (R Bioconductor version 3.9). We implemented the code into the R package “sigident” (https://gitlab.miracum.org/clearly/sigident
), which provides the four main functions—sigidentDEG
, and sigidentPrognostic.
The whole workflow is documented in detail in the R package vignette.
Supplementary Table S5
lists the used R packages. The newly created “sigident” R package integrates a (i) meta-analysis (multiple dataset integration), (ii) functional gene expression analysis, and (iii) the calculation of statistically robust multi-gene signature combinations. As an application example, we used lung cancer datasets from the GEO database (GSE18842, GSE19804, and GSE19188). After merging, we divided the dataset into a training (80%) and test (20%) dataset for the calculation of the diagnostic signature. Moreover, we validated the diagnostic signature in three independent datasets (GSE30219, GSE102287, GSE33356). For the prognostic signature, we performed a survival analysis using the GSE19188 which includes survival information and validated the signature in two independent datasets (GSE30219, GSE50081).
For the meta-analysis (dataset download, normalization, merging) and the functional gene expression analysis (analysis for DEGs, heatmap), we used our previously published sample merging approach, which is based on a modified code of the inSilicoMerging package combined with the limma package [27
]. This approach has been developed further in order to integrate it into the “sigident” R package framework. In detail, it uses the R package GEOquery version 2.52.0 for dataset downloading [48
], gcrma package version 2.56.0 for CEL file loading, background correction, quantile normalization, and log2-transformation [28
], Biobase package version 2.44.0 for integration of standardized data structures [13
], gplots package version 188.8.131.52 for graphical representation [49
], and the limma package version 3.40.6 for the DEG analysis [50
]. We extended the code by detecting batch effects using a guided principal component analysis from the gPCA package version 1.0 [51
]. For batch effect correction, we used empirical Bayes framework applying the ComBat function from the sva package version 3.32.1 [52
] considering different groups (tumor, ctrl). As a DEG analysis is known to generate false positive results [36
], we applied a multiple-testing correction using the Benjamini and Hochberg approach to control the FDR [53
]. We used a stringent q-value (adjusted FDR value) of 0.05.
Furthermore, for the DEGs we added a functional gene ontology (GO) and KEGG pathway enrichment analysis using the goana and kegga functions from the limma package (Entrez IDs as input). A further GO and pathway over-representation test is implemented using the clusterProfiler package version 3.12.0 [54
] (including FDR control, DEGs are mapped to their Entrez-IDs as input), whereas specific pathways can be further investigated using the pathview package version 1.24.0 [55
The calculation of statistically robust multi-gene signature combinations focuses on diagnostic and prognostic signatures. For diagnostic signatures, we used the LASSO and Elastic net penalty as implemented in the R package glmnet version 2.0.18 [56
]. The hyper-parameter alpha can manually be set to a value between 0 and 1 or can automatically be calculated in combination with the tuning parameter lambda based on cross-validation and a grid search applying the wrapper function train as implemented in the caret package version 6.0.84 [57
]. In the case of a fixed value for alpha, lambda is determined by 10-fold cross-validation, and a leave-one-out cross-validation is also possible. For calculation of the Receiver Operating Characteristics (ROC) and the Area Under the Curve (AUC) value of the ML models we used the pROC package version 1.15.3 [58
For the prognostic signature detection we applied a survival and risk assessment analysis using a Cox Proportional Hazard Model as implemented in the survival R package version 184.108.40.206 [59
]. The Cox Proportional Hazard regression analysis identifies genes that have a significant effect size on the survival outcome. To generate a prognostic signature, we applied a classification algorithm that assigns patients in high and low risk groups based on the expression profiles of the identified survival correlated genes between tumor and non-tumor samples. Survival curves were plotted using the survminer package version 0.4.5 [60