Exploration of Machine Learning Models for Prediction of Gene Electrotransfer Treatment Outcomes

Abstract

Gene electrotransfer (GET) is a physical method of gene delivery to various tissues utilizing pulsed electric fields to transiently permeabilize cell membranes to allow for genetic material transfer and expression. Optimal pulsing parameters dictate gene transfer efficiency and cell survival, which are critical for the wide adaptation of GET as a gene therapy technique. Tissue heterogeneity complicates the delivery process, requiring the extensive optimization of pulsing protocols currently empirically optimized. These experiments are time-consuming and resource-intensive, requiring large numbers of animals for in vivo optimization. Advances in machine learning (ML) and computing power, data analysis, and model generation using ML techniques, such as neural networks, enable predictive modeling for GET. ML models have been used previously to predict ablation performance in irreversible electroporation procedures and single-cell electroporation platforms. In this work, we present ML predictive models that could be used to optimize pulsing parameters based on already completed experiments. The models were trained on 132 data points from 19 papers with the Matlab Statistics and Machine Learning Toolbox. An artificial neural network (ANN) was generated that could predict binary treatment outcomes with an accuracy of 71.8%. Support vector machines (SVMs) using selected features based on ${\chi }^2$ tests were also explored. All models used a maximum of 24 features as input, spread across target species, needle configuration, pulsing parameters, and plasmid parameters. Pulse voltage and pulse width dominated as the critical parameters, followed by field strength, dose, and electrode with the greatest impact on GET efficiency. This study elucidates areas where predictive ML algorithms may ideally inform GET study design to accelerate optimization and improve efficiencies upon the further training of these models.

1. Introduction

Gene transfer, and the resulting expression of the encoded protein, has been used for decades in the treatment of disease, as well as for improvements in quality of life for those with genetic disorders [1]. There are several well-known methods for the delivery of genetic material: viral, physical, and nanoparticle-based. In the domain of physical transfection technologies, gene electrotransfer (GET) has become popular as an effective and relatively safe way to deliver genetic material to target tissues [2]. GET involves the use of pulsed electric fields (PEFs) to facilitate the transfer of exogenous genetic material, typically naked plasmid DNA, through the cell membrane to be translated by the host cell into a therapeutically useful protein [3,4]. A typical treatment involves the injection of the plasmid DNA, followed by the direct application of the prescribed PEFs to the desired tissue. The actual shape, magnitude, repetition rate, and count of these PEFs varies dependent on the plasmid DNA used, tissue to be treated, and physical configuration of the electrode used to deliver them [5,6].

Gene transfer efficiency and resulting protein expression are highly dependent on selecting proper pulsing parameters and resulting electric fields [6]. Applying GET to various tissues in the body has also shown that expression is highly dependent on intrinsic tissue properties and, thus, no universal approach exists to date [7]. These parameters, both electrical and biological, have been the focus of previous, empirical optimization efforts.

In the case of electrical parameters, the number of pulses, pulse width, applied voltage, repetition rate, and electrode configuration determine the magnitude, direction, and distribution of electric fields in the treated tissue. By changing any of these parameters, the amount and distribution of electrical energy delivered to the tissue can be modulated. These effects have been shown by modeling the underlying processes through the finite element modeling (FEM) of tissues, pulses, and electrode configurations [8,9,10], and the impact of changing these parameters has been shown in empirical optimization studies [11,12].

There is a delicate balance between efficiently and transiently permeabilizing cells within tissues and overpulsing [13]. Overpulsing leads to permanent damage to cell membranes resulting in cell death, which is undesirable for gene therapy applications and enters the field of irreversible electroporation (IRE) or electrical ablation. Underpulsing, on the other hand, results in reduced cell permeabilization and insufficient or zero gene transfer efficiency [6,14,15].

Electrical parameters are not the only variables impacting the efficacy of gene electrotransfer success. Biological parameters such as tissue characteristics and the size of the plasmid DNA used can impact treatment success. Specific tissues under treatment, such as skin, muscles, or tumors, all have varying electrical properties that can greatly impact the distribution of electric fields during treatment [16]. Similarly, tissue impedance was found to change as a GET treatment progressed [17,18,19,20], and these changes were not identical across all tissue types [21]. Thus, electric field distributions change as GET treatment progresses, depending on the underlying tissue type. Regarding the size of the plasmid DNA used, plasmid DNA with larger backbones was found to decrease transfection efficiency when compared to plasmid DNA with smaller backbones [5]. So, despite ultimately expressing the same protein, the smaller plasmid DNA allowed for higher transfection efficiency despite the same electrical parameters being used.

Since the first treatments in the 1980s, the optimization of GET protocols has been an experimentally intensive endeavor. These experiments require large numbers of animals, usually mice, rats, and guinea pigs, but also validation in large animals such as pigs prior to progression to clinical trials. Despite this, successful GET protocols have been established in various tissues empirically, including the development of GET-mediated interleukin-12 (IL-12) delivery to melanoma with successful phase I and phase II clinical trials, indicating that plasmid DNA delivery encoding IL-12 can stimulate the immune system to clear treated lesions and metastatic disease [22,23]. While there have been major advancements in pulse type, electrode design [24,25,26], external stimuli such as heat [27,28], and real-time treatment endpoint detection using impedance [20,29,30], the underlying problem of searching the vast parameter space remains.

Machine learning models have been in use for many years but, due to the recent increases in computing power, have found more mainstream accessibility and acceptance. Models have been applied to diagnostic image processing in radiology [31] and histological imaging [32] and in adverse event detection using respiratory data [33]. Machine learning models have also been applied to the field of electroporation, specifically artificial neural networks (ANNs) for IRE [34,35], support vector machines (SVMs) for interpreting post-ablation outcomes from magnetic resonance imaging (MRI) [36], and fully convolutional networks (FCNs) and computer vision for the identification of potential tumors to treat using a multi-electrode structure [37]. In the field of GET, machine learning has only been used to locate cells using computer vision for the application of targeted, single-cell therapies [38] or in in vitro studies limited to single cell lines [39]. To date, the prospect of applying these techniques to GET on a tissue scale remains elusive.

A complicating factor to the development and deployment of such a machine learning model is that there is no standard for the reporting of pulsing parameters across different studies and no existing repository or database for documenting pulsing parameters and resulting tissue responses. While the methodologies used in reporting are varied, the related parameters can often be translated from one lab to another, such as how some labs report the time between pulses or a pulse frequency, which both effectively describe the same thing. However, this introduces the possibility of errors in translation or misinterpretation of the intention behind the way in which these values are reported. A standardized method would avoid such potential errors and allow for the rapid creation of datasets where all necessary parameters are reported in a uniform way, avoiding the need for translation or interpretation.

ANNs were the first choice for the models as they are extremely customizable and flexible when dealing with variable input data types and have been successfully employed in similar endeavors in the field of IRE [34,35]. However, they suffer from a need for large datasets to effectively predict outcomes. This need grows exponentially as the number of features, or data categories, increases [40]. Since the data available are relatively limited and have a comparatively large number of features, SVMs were chosen as a second option as they are capable of accurate predictions with more limited sample sizes compared to ANNs [41].

This paper presents an exploratory study on the application of two machine learning models to a dataset of GET experiments published over the course of the last 30 years.

2. Materials and Methods

2.1. Paper Selection and Data Curation

The initial literature search for papers for inclusion in the dataset was limited based on whether the papers

• conducted GET experiments with PEFs and plasmid DNA;
• did not use additional therapeutic techniques to achieve success;
• used a pulse configuration with a maximum of two separate voltage levels;
• provided results in terms of significant difference from the control;
• conducted experiments either in vitro or in vivo.

The data from the papers were then manually captured in a spreadsheet using the features described in

Table 1. Table of features from each of the source papers and their descriptions.

Feature	Description
Common Name	Common name of plasmid used in study
Dose	Dose of plasmid in μg
Electrode	Electrode configuration
Electrode Type	Penetrating or non-penetrating electrode
Backbone Name	Name of backbone in plasmid
Promoter Name	Name of promoter in plasmid
Construct Size	Size of entire plasmid in kb
Backbone Size	Size of backbone of plasmid in kb
Insert Size	Size of insert of plasmid in kb
Encoding For	Name of protein encoded for
Pulse Technique	Straight pulsing between 2 electrodes or multielectrode array pulsing
Pulse Polarity	Same or alternating polarity when different pulse parameters are used
Pulse V1/V2	Pulse amplitude of the first/second pulse in volts
Field Strength 1/2	Field strength of the first/second pulse in volts/cm
Pulse Width 1/2	Pulse width of the first/second pulse in μs
Number of Pulses	Number of pulses/pulse trains
Dwell Time	Time between pulses/pulse trains in s
Electrode Spacing	Distance between electrodes in cm
General/Specific Tissue	Target tissue for the study
Sex	Sex of the animal/source cells
General/Specific Species	Species of the animal/source cells
Expression Location	Location(s) of expression determined via histology of the target tissue(s)
Outcome	Success or failure of the treatment based on source author(s) significance criteria

.

For training inputs, electrode configurations and electrode types were labeled based on the descriptions provided in the source papers. Electrode types were split into either penetrating or non-penetrating electrodes, where penetrating electrodes were intended to pierce the target tissue and non-penetrating electrodes were intended to rest on the surface. In vitro studies were classed as non-penetrating electrodes due to the use of cuvettes and plate electrodes in their methodologies.

The features Specific Tissue and Tissue as well as Specific Species and Species were added to avoid overspecification of the experimental targets. The inclusion of both features allowed the learning algorithms to decide if a particular outcome was better predicted by a general species/tissue designation rather than the specific tissue or species under study (e.g., ‘Tumor’ vs. ‘B16 Melanoma’ or ‘Rat’ vs. ‘Sprague Dawley Rat’).

It was often the case that either the voltage or the field strength was not directly reported. However, if the paper included the relevant electrode spacings, the missing value was calculated using

E = V/d,

(1)

where E is the electric field strength in V/cm, V is the applied voltage in V, and d is the electrode spacing in cm. While this does not provide an accurate representation of the field strength between electrodes, it is commonly used in the field to relate the applied voltage and expected field strengths when reported in the literature [25,42,43,44].

In the case of the monopolar electrodes [45], the method of determining the field strength directly from the parameters and geometries reported in the papers was not possible as the grounding electrode was significantly distant from the active electrode. In this case, COMSOL FEM simulations were employed to simulate the field strengths proximal to the active electrode during pulsing. Geometries and tissue types were determined from the papers, whereas the electrical properties of the tissues of interest were determined based on electrical parameters described in [16].

Sex was handled using four potential variables: Male, Female, M/F, and NP. M/F was used in cases where the group under study was a mix of male and female animals but the results and conclusions were not separated by sex. NP was used for one study where the sex of the animals was not provided. Sex and species for in vitro studies were entered based on the sex and species of the source of the cells under study.

In cases where the pulse parameters denoted only one voltage, all the dual features (V1/V2, Field Strength 1/2, etc.) were filled with the same information. For studies where the pulse parameters did vary within a pulse train, the first set of parameters always corresponded to the parameters of the first pulse in the train. All voltages and field strengths were entered as positive values, with pulse polarity accounting for any positive to negative voltage transitions. In cases where pulse frequency was reported instead of the time between pulses, the time between pulses was calculated as the inverse of the pulse frequency.

The outcome of each referenced study varied depending on both the goal of the study and the plasmid used to achieve said goal. These outcomes ranged from the amount of protein expression to patient outcomes in clinical trials. Due to the variability of the biological responses reported in the literature, the outcomes of the studies were entered as a binary variable rather than entering the outcome data from the papers directly. This binary classification allowed the models to report either a ‘success’ or ‘fail’, rather than dynamically predicting gene transfection efficiencies or patient outcomes based on the goal of the study. In the case of human cancer trials, success was considered as either stable disease or a complete response, whereas failure was considered in the case of progressive disease. In all other cases, success was considered a statistically significant improvement in the biological responses between the test group and a fixed control. In the event that a particular paper had multiple reported experiments, of which only a portion conformed to the requirements for inclusion in the dataset, only the relevant experiments were included in the dataset.

2.2. Determination of the Most Important Features

${\chi }^2$ tests were conducted between each feature and the binary outcome variable. The features were then sorted by the respective p-value, with lower p-values being ranked more highly. SVMs were trained on limited feature sets of 5, 6, 7, 8, and 9 features, informed by the results of the ${\chi }^2$ tests, to determine if using limited feature sets improved model outcomes. The process of training ANNs should include weighing features with the greatest impact on the outcome, but training with the limited datasets was performed with the ANNs as well for exploratory purposes. Reducing the number of features available to the ANN models had the potential to offset the need for the large datasets discussed in Section 2.2.

To validate the ranking of predictor variables, the above ${\chi }^2$ tests were carried out 100 times with randomized subsets of data containing 70% of the source dataset. The resulting rankings were then plotted with 95% confidence intervals to determine if the rankings were independent of individual samples.

2.3. Training of the Machine Learning Models

The 132 experiments were split into two datasets: one for training and one for testing; 70% of the data was used in training, while the remaining 30% was used in the test set. The ratio between significant and non-significant outcomes was maintained in both the training and test sets.

In the case of the SVM models, six total models were trained. The ‘basic’ SVM model was trained on a subset of data containing all predictors. The remaining five SVM models were trained on a further limited subset of the data containing the top 5, 6, 7, 8, and 9 predictors. The top predictors were determined using the ${\chi }^2$ tests described in Section 2.2.

Eight ANN models were trained targeting different optimization techniques. The first model was trained using a default ANN, with one layer of width 10, a rectified linear unit (ReLU) activation function, and standardized feature inputs. A second ANN was trained by adjusting the regularization strength, $\lambda$, while minimizing cross-validation classification error. The value of $\lambda$ corresponding to the minimum cross-validation classification error was then used in the final model. The third ANN was subjected to an expected improvement in the Bayesian optimization procedure to discover optimal parameters, while avoiding overfitting the model to the training data. The optimizer was allowed to adjust layer number, layer width, activation function, and regularization strength. The ‘basic’, ‘$\lambda$-optimized’, and ‘hyperparameter-optimized’ ANN models were trained with access to all predictors in the 70% training set. The remaining five ANN models were trained using the previously mentioned default parameters but on limited datasets of the 5, 6, 7, 8, and 9 most important features as determined by the ${\chi }^2$ testing described in Section 2.3.

2.4. Assessing the Accuracy of the Models

In all cases, a confusion matrix was generated for each of the 14 models comparing predicted outcomes from the models to the true labeled data from the test dataset. Several figures of merit were calculated based on the confusion matrix [46]: accuracy, sensitivity, specificity, F₁-score, Matthews correlation coefficient (MCC), and receiver operator characteristic (ROC) area under the curve (AUC).

Accuracy is effectively a measure of generally how often a model is correct in its predictions considering both positive and negative predictions. Accuracy was determined as the ratio of the sum of true positive (TP) and true negative (TN) predictions to the total number of samples in the test set. Sensitivity is a measure of the accuracy of a model only regarding positive predictions and was calculated as the ratio of the true positive predictions to the total number of actual positives in the data. Specificity is the same as sensitivity, except only regarding negative predictions. It was calculated in a similar manner as the ratio of true negative predictions to the total number of actual positives in the data. The calculations are summarized below:

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}}},$$

(2)

$$\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}},$$

(3)

$$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}}},$$

(4)

Two general goodness-of-fit measures were considered for this study: F₁-score and MCC. The F₁-score is a harmonic average of precision and recall, while the MCC is a more robust measure of classification models trained on unbalanced datasets, where the two classifications are not equally represented [46]. In this case, the data contain a larger proportion of significant results versus non-significant results. The F₁-score ranges from 0 to 1 (0 being worst), with a value of 0.5 being equal in effectiveness to a coin-flip predictor. The MCC score ranges from −1 to 1 (−1 being worst), with an MCC of 0.3 being considered acceptable and an MCC of 0 representing the coin-flip predictor [46]. The two measures were calculated as follows:

$${\mathrm{F}}_1={\displaystyle \frac{2\mathrm{T}\mathrm{P}}{2\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}},$$

(5)

$$\mathrm{M}\mathrm{C}\mathrm{C}={\displaystyle \frac{\mathrm{T}\mathrm{P}\cdot \mathrm{T}\mathrm{N}-\mathrm{F}\mathrm{P}\cdot \mathrm{F}\mathrm{N}}{\sqrt{\left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}\right)\cdot \left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}\right)\cdot \left(\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}\right)\cdot \left(\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}\right)}}},$$

(6)

Receiver operator characteristic (ROC) curves were also generated, and the area under the curve (AUC) was calculated as an additional measure to determine suitability for the models. An ROC AUC of greater than 0.7 was considered acceptable and values greater than 0.8 were considered good. A value near 0.5 was considered similar to a coin-flip classifier [47].

3. Results

3.1. Dataset Summary

A total of 19 studies encompassing 132 experiments were included in the dataset. A summary of the sources is provided in

Table 2. Source papers with selection of features from each group of experiments showcasing the variability in existing GET studies.

Source	Plasmid Common Name	Species/Cell Line	Electrode	Pulse Polarity	Experiments
[42]	pRc/CMV-Luc	Sprague Dawley Rat	6-Needle Circle	Positive	7
[48]	pUT531	B6D2F1 Mouse	2-Plate	Positive	10
[43]	pCMV-Luc+	Sprague Dawley Rat	7-Needle	Positive	11
[49]	pCMV-Luc+	BC57B1/6 Mouse	7-Needle	Positive	1
[49]	pCMV-Luc+	BC57B1/6 Mouse	7-Needle	Positive	4
[50]	pLuc	SKH1 Mouse	Caliper	Alternating	12
[25]	pCMVLuc	Wistar Rat	Caliper	Positive	7
[51]	pVEGF	Sprague Dawley Rat	4-Plate	Positive	2
[52]	gWiz-Luc	C57B1/6 Mouse	Caliper + 6-Needle	Positive	8
[22]	pUMVC3-hIL-12-NGVL3	Human	6-Needle Circle	Positive	24
[53]	pEGFP-C1	Chinese Hamster Ovarian Cells	4-Pin Circle	Positive + Alternating	4
[54]	phGFP-S655T	C57Black/C Mouse	2-Plate	Alternating	1
[54]	phGFP-S655T	Wistar Rat	2-Plate	Alternating	1
[26]	gWiz-Luc	HaCaT Cells	Cuvette	Positive	2
[44]	pCMV-tdTomato	C57Bl/6 Mouse	Contact Wire	Positive	9
[11]	pDRIVE2-CMV	Mesenchymal Stem Cells	Cuvette	Positive	7
[55]	pVAX2-LUC	C57BL/6 Mouse	Plate + 7-Pin Hex	Positive	3
[56]	pVax1-hVEGF165(pVEGF-A)	Yorkshire Pig	4-Pin	Positive	1
[3]	pCH110	Wistar Rat	2-Pin	Positive	4
[45]	gWiz-Luc and NTC9385R-Luc	Sprague Dawley Rat	Monopolar + 4-Pin	Positive + Alternating	4
[5]	gWiz-Luc and NTC9385R-Luc	Sprague Dawley Rat	Caliper	Alternating	10

. Approximately 60% of experiments in the dataset were considered successful by the source.

3.2. FEM Simulation of Monopolar Electrode

The results of the COMSOL simulation are shown below in Figure 1 during the 90 V, 100 ms pulses from [45]. These results showcase the high variability in electric field strengths in tissues, even when uniform voltages are applied. The electrical parameters used in the simulated muscle were taken from [16].

3.3. ${\chi }^2$ Results for Feature Selection

The results of the ${\chi }^2$ test are provided in Figure 2. The highest-ranked features were pulse voltage and pulse width, which means that these variables were most highly correlated with the outcome variable, either the ‘success’ or ‘failure’ of the treatment. Scores are given as −log(p), where p is the p-value of the corresponding ${\chi }^2$ test between the individual feature and the outcome of the experiments. Scores greater than 1 were considered to significantly contribute to the prediction of the outcome variable [57]. The differences between subsequent scores decreased after the highest four scores, so those scores were always included as part of the limited predictor dataset.

The bootstrapped rankings of 100 resamplings are shown in Figure 3. While several predictors changed relative ranks compared to the single-sample ${\chi }^2$ testing described in Figure 2, the top four predictors remained the same, making them robust to the resampling of the data.

3.4. Model Results

All models generated performed similarly in terms of accuracy, with feature-limited models outperforming their basic and parameter-optimized variants and accuracies ranging from 53% to 72%. Model performance began to differentiate when looking at the more specific figures of merit. The sensitivity and F1-scores of the Basic ANN, $\lambda$-Optimized ANN, and Basic SVM were particularly poor in performance, and the MCC of the Basic ANN suggested that it was only marginally better than a coin flip.

Table 3. Figures of merit for each of the 14 machine learning models generated.

Model	Accuracy	Sensitivity	Specificity	F1-Score	MCC	ROC AUC
Basic ANN	59.0%	0%	100.0%	0.0	−1.0	0.6036
$\mathsf{\lambda }$ Optimized ANN	66.7%	56.3%	73.9%	0.5806	0.3050	0.6536
Hyperparameter Optimized ANN	59.0%	43.8%	69.6%	0.4667	0.1365	0.4875
Top 5 Features ANN	71.8%	43.8%	91.3%	0.5600	0.4092	0.4679
Top 6 Features ANN	64.1%	25.0%	91.3%	0.3636	0.2223	0.8478
Top 7 Features ANN	69.2%	43.8%	87.0%	0.5385	0.3459	0.8288
Top 8 Features ANN	66.7%	31.3%	91.3%	0.4348	0.2891	0.8274
Top 9 Features ANN	69.2%	37.5%	91.3%	0.5000	0.3509	0.8333
Basic SVM	53.9%	31.3%	69.6%	0.3571	0.0087	0.8125
Top 5 Features SVM	59.0%	0%	100.0%	0.0	−1.0	0.4214
Top 6 Features SVM	59.0%	0%	100.0%	0.0	−1.0	0.3234
Top 7 Features SVM	64.1%	31.3%	87.0%	0.4167	0.2218	0.6012
Top 8 Features SVM	64.1%	31.3%	87.0%	0.4167	0.2218	0.6012
Top 9 Features SVM	64.1%	31.3%	87.0%	0.4167	0.2218	0.5744

summarizes the figures of merit for all models. Figure 4 provides the CMs and Figure 5 provides the ROC curves for the top four ANN models by ROC AUC: 6–9 top feature ANNs. All performance metrics were calculated with the original test set, without re-randomizing the data, thereby avoiding possible inflated performance by including a subset of training data.

While the basic SVM outperformed the basic ANN, restricting features fed into the ANN resulted in a marked jump in performance in comparison with even the basic SVM. Also of note is that all models had drastically different results for specificity and sensitivity, with specificity being markedly higher in all cases, regardless of overall performance. The similar results for the top 6–9 ANN models also reaffirmed that adding further features past the top 6 based on the ${\mathsf{\chi }}^2$ results in Section 3.3 did not improve model performance.

4. Discussion

To date, GET optimization requires the extensive use of costly cell cultures and animals to refine ideal PEF conditions and associated parameters to maximize expression with minimal side effects. A typical new GET study will iterate on the variables presented in Table 2 to achieve a set of working conditions that achieve the desired outcome. These myriad parameters are typically informed only by lab history itself, with highly varied preferences between labs. A model that is informed by a large body of data from multiple labs has the potential to enable physical studies to start from a more advantageous state and reduce the need for costly materials and lab time. These models narrow down the need for parameters that need to be tested empirically in animals for specific tissues with specific electrode configurations. Ultimately, this would reduce the time to clinical translation and get these new parameters into the hands of practicing clinicians sooner. This work presents a first foray into developing such ML algorithms to inform GET working conditions to reduce the variability, cost, and complexity of studies and improve overall study results with higher fidelity GET optimization informed by a trained predictive model.

As a first step, basic ANN and SVM models were trained on the dataset without any feature reduction, with the results presented in Table 3. While the SVM model performed better than the ANN model, both models performed poorly. The accuracies, sensitivities, and specificities for both models were all below 70% and the MCCs for both models were about or below the value expected for a coin-flip predictor. The Basic SVM model did have an ‘acceptable’ AUC of >0.8, but, given the poor performance in other metrics, this metric on its own was not considered acceptable.

In the training of ANNs, there are a multitude of parameters to adjust that may increase the performance of the underlying model. The first of these is the regularization strength, $\lambda$. This parameter can be adjusted by iterating the underlying model under different values of $\lambda$ and choosing the model that reduces error during validation. The optimized model had better performance than the Basic ANN, as shown in Table 3, but still performed relatively poorly when compared to desired thresholds. These were not unexpected results, as most machine learning models require large datasets to begin to achieve higher levels of accuracy and performance. ANNs specifically scale the required dataset size based on the number of features used to describe each data point. So, the more variables that are used to describe the underlying data, the more data points are required for an ANN to approach usable metrics.

To reduce the effect of the relatively small sample size, the number of features used to train the models was reduced using the results of a bootstrapped ${\chi }^2$ test. The results of one iteration of this test are provided in Figure 2, which shows that voltage and pulse width were amongst the features that most strongly predicted the outcome. This outcome follows prevailing theories in the field, where labs will generally start by adjusting physical pulse parameters when attempting to optimize a procedure. However, due to variability in the data, a bootstrapped version of the test was completed, where the data were resampled 100 times and 95% confidence intervals were plotted for each of the ranks. These data are summarized in Figure 3. Even after bootstrapping, these results show that, in the available data, pulse and electrode design were the most impactful features when it came to successful outcomes, reinforcing the explainability of the underlying ‘logic’ of the machine learning models.

Using the results of the ${\chi }^2$ analysis, reduced-feature-set datasets were constructed using the top 5, 6, 7, 8, and 9 features. The minimum feature set of five was determined as the threshold in the ${\chi }^2$ results where the subsequent features did not have a large difference in rank score from the previous features in Figure 2. These reduced-feature datasets were used to train ANNs and SVMs, with the resulting metrics tabulated in Table 3. As expected, the models trained on reduced features had higher performance than the full-feature datasets. Four out of five models had an AUC > 0.8 and three of the five also had improved MCCs > 0.3. The ROC curves and truth tables of these selected models are provided in Figure 4, with their performance metrics presented in Table 3 alongside those of the other models.

The SVM versions of the reduced-feature models did not see such a boost in performance as observed with the ANN models. In most cases, the performance was reduced compared to the full-feature model. This result was likely due to the reduced features the SVM had available to train the model. While ANNs act as if they have a ‘logic’ to their decisions, SVMs create support vectors from the available training data to separate the outcomes in a higher dimensional space. Reducing the feature size reduces the available variables for the SVM to create these support vectors, which is likely why the reduced-feature SVM models lost performance compared to the ANN models.

Ultimately, the goal of these models is to inform researchers and clinicians of the potential of a proposed set of parameters before spending resources to investigate them directly. However, as shown in the collected dataset, the field at large operates on a wide range of species and tissue types and uses a variety of plasmid DNA, depending on the application. The wide variability in application and subject tissue presents a limitation in model generalizability. While these models would perform well given the tissues, species, or electrodes that the models were trained on, the introduction of a tissue or electrode unknown to the model would likely not provide similar results to those presented here. A solution to this generalization problem is to expand the electrode, tissue, and species variables beyond a simple label. FEM simulations of electric field strengths between the electrodes would alleviate part of this unknown by allowing for the entry of electrode and tissue parameters directly into both the FEM and machine learning models, effectively specifying electrodes by their field distributions and tissues by their electrical parameters rather than by labels. This one step would eliminate a large portion of the limitation in the generalizability of these models, but would take a concerted effort by the community at large to achieve.

Finally, globally, there is a notable limitation of this study due to inherent variabilities from lab to lab and even from operators within labs in how experiments are designed and carried out, how data are reported, and how field strengths are determined. The variability of GET test equipment and the sensitivity of the analyses are inherent in such an analysis as there is a lack of standardization in the hardware, software, test methods, and reporting, as we noted in reviewing 19 different papers for this model. Field strengths in particular are often reported in a simplified manner using Equation (1). This can be an oversimplification, as shown in Figure 1, where the field strengths and thus voltage distributions are anything but uniform. Furthermore, it is not typical for labs to report failed experiments unless they are relevant in the context of reporting successful treatment, such as in the case of a known negative control. This leads to lopsided datasets that are full of successes, with very few failures. Even though this dataset ended up with a 60–40% split of successes vs. failures, the majority of the ‘failures’ were, in fact, control data that were known to produce either an insignificant effect or no effect whatsoever. This leads to a false uniformity in the conditions leading to the failure of treatment and resulting disparities between sensitivity and specificity metrics. Publishing, or at least sharing, unsuccessful testing results will not only provide additional data points to train these models on but also add variability to unsuccessful conditions, which is not present in the current literature.

In creating a robust model, all data should be considered, regardless of the success or failure of the attempted treatment. Furthermore, efforts to standardize and monitor the pulsing, delivered PEF, its conditions, and controlled and standardized analyses are crucial to training and an area of further research for our group. The optimal solution to standardization is for all experiments to come with accompanying FEM simulation data, captured pulse waveforms, and histology in the treatment area. These data could then be fed as images or multidimensional arrays into a machine learning model to more accurately inform the model of real spatial and temporal variations in field strength during treatment and correlate those variable field strengths to effective transfection in the underlying tissue. These data would also help alleviate lab-to-lab variability as, even if labs define treatment parameters differently, standardized values could be extracted from the provided data.

5. Conclusions

This work presented a first set of ML analyses from 132 data points gathered from 19 published papers to generate algorithms for optimizing GET conditions in silico. This initial training highlighted the importance of pulse voltage and pulse width (added values of successful pulse parameters), followed distantly by field strength, dose, and electrode configuration as guiding design principles for effective GET. The initial results show that it is possible to generate an ML model to inform optimal GET parameters to reduce the required time, cost, and animal numbers in future experiments. While the initial ANN models performed well, improvements will require the contribution of additional GET data including any unsuccessful treatments, as well as the more accurate modeling of real electrical parameters in the tissues and cells under study.