Next Article in Journal
Imaging Modality to Guide Left Atrial Appendage Closure: Current Status and Future Perspectives
Next Article in Special Issue
Advances in Therapies to Treat Neonatal Hypoxic-Ischemic Encephalopathy
Previous Article in Journal
Role of Dual-Acquisition Noninvasive Cardiac CT Imaging for the Detection of Vasospastic Angina
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JCM
Volume 12
Issue 11
10.3390/jcm12113755
jcm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Artificial Intelligence and Machine Learning in the Diagnosis and Management of Stroke: A Narrative Review of United States Food and Drug Administration-Approved Technologies

by
Anirudha S. Chandrabhatla
1,2,
Elyse A. Kuo
1,2,
Jennifer D. Sokolowski
2,
Ryan T. Kellogg
2,
Min Park
2 and
Panagiotis Mastorakos
2,3,*
1
School of Medicine, University of Virginia Health Sciences Center, 1215 Lee Street, Charlottesville, VA 22903, USA
2
Department of Neurological Surgery, University of Virginia Health Sciences Center, 1215 Lee Street, Charlottesville, VA 22903, USA
3
Department of Neurological Surgery, Thomas Jefferson University Hospital, 111 S 11th Street, Philadelphia, PA 19107, USA
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2023, 12(11), 3755; https://doi.org/10.3390/jcm12113755
Submission received: 30 April 2023 / Revised: 22 May 2023 / Accepted: 26 May 2023 / Published: 30 May 2023
(This article belongs to the Special Issue Clinical Aspects of Cerebral Ischemia)
Browse Figures
Versions Notes

Abstract

:
Stroke is an emergency in which delays in treatment can lead to significant loss of neurological function and be fatal. Technologies that increase the speed and accuracy of stroke diagnosis or assist in post-stroke rehabilitation can improve patient outcomes. No resource exists that comprehensively assesses artificial intelligence/machine learning (AI/ML)-enabled technologies indicated for the management of ischemic and hemorrhagic stroke. We queried a United States Food and Drug Administration (FDA) database, along with PubMed and private company websites, to identify the recent literature assessing the clinical performance of FDA-approved AI/ML-enabled technologies. The FDA has approved 22 AI/ML-enabled technologies that triage brain imaging for more immediate diagnosis or promote post-stroke neurological/functional recovery. Technologies that assist with diagnosis predominantly use convolutional neural networks to identify abnormal brain images (e.g., CT perfusion). These technologies perform comparably to neuroradiologists, improve clinical workflows (e.g., time from scan acquisition to reading), and improve patient outcomes (e.g., days spent in the neurological ICU). Two devices are indicated for post-stroke rehabilitation by leveraging neuromodulation techniques. Multiple FDA-approved technologies exist that can help clinicians better diagnose and manage stroke. This review summarizes the most up-to-date literature regarding the functionality, performance, and utility of these technologies so clinicians can make informed decisions when using them in practice.

1. Introduction

Stroke is a neurological emergency and the fifth leading cause of death in the United States [1,2,3]. Established clinical interventions exist for many stroke subtypes, such as large vessel occlusion (LVO) and intracranial hemorrhage (ICH). Prompt treatment is one of the more important factors in maximizing the preservation of neurological function. Notably, each minute of treatment delay results in significant neuronal death and the loss of 4.2 days of healthy life [4].
Tools to improve the speed and accuracy of stroke diagnosis and treatment could improve patient outcomes. Artificial intelligence/machine learning (AI/ML) will play a large role in developing such tools. AI/ML in healthcare is growing at 40% per year, and its adoption has the potential to cut USD 150 billion in healthcare costs by 2026 [5]. Recognizing the potential AI/ML has to improve healthcare, the United States Food and Drug Administration (FDA) has developed new protocols to assess the safety and efficacy of AI/ML-enabled health technologies [6]. AI/ML-enabled algorithms have been leveraged for various clinical applications such as detecting liver fibrosis [7], analyzing EKGs [8], monitoring Parkinson’s [9], diagnosing glaucoma [10], and classifying lung cancer [11]. The FDA has approved 22 AI/ML-enabled technologies for indications specifically related to stroke diagnosis and rehabilitation. Existing literature reviews in this area have broadly evaluated AI/ML algorithms that have largely been developed for research purposes [12,13,14,15,16]. No study to date has comprehensively evaluated the real-world clinical performance of clinically available, FDA-approved devices indicated for the diagnosis and management of stroke. This review aims to synthesize the most relevant, up-to-date information related to these technologies and provide an overview of their unique functionalities and performances regarding improving clinical workflows and outcomes.

2. Methods

2.1. Technology Search

We sought to identify all FDA-approved, AI/ML-enabled medical technologies with indications for ischemic stroke and/or ICH. To compile this list, our search had two components.
First, we examined the previously cited [17] database that is directly maintained by the FDA and contains 343 AI/ML-enabled technologies that the FDA has approved. We extracted technologies labeled “radiology” and “neurology” (n = 253). As previously described in the literature, this list does not have search or filter functionality to assess technology descriptions or approval letters [18]. Therefore, two reviewers analyzed the 253 official FDA approval letters and/or company websites to determine their relevance to ischemic stroke/ICH. This search resulted in 30 technologies, 9 of which were listed for multiple indications, resulting in 21 unique technologies (Figure 1).
Second, we conducted an internet search in accordance with previously described methods [19] to identify technologies with approval statuses not covered in the FDA list. In short, we conducted a Google search with the compound search term: (“stroke” OR “intracerebral hemorrhage” OR “CVA”) AND (“artificial intelligence” OR ”ai” OR ”machine learning” OR ”ml” OR ”deep learning”) AND (“FDA approved” OR “FDA approves” OR “FDA approval”) (Figure 1). The search returned 82,600 results, and URLs were sequentially assessed for information regarding AI/ML-enabled, FDA-approved technologies. As per previously described stopping criteria [19], we concluded the search after 40 new results (i.e., 4 full pages of Google results) failed to reveal new technologies. We evaluated 70 links; 18 links included relevant technologies, 7 of which were unique (Figure 1).
Combining the results from both search components, we assessed 28 ischemic stroke-/ICH-related technologies. After excluding 6 duplicates, we arrived at a total of 22 unique technologies (Figure 1).

2.2. Literature Search

We queried PubMed and relevant company websites to assess the most up-to-date (post-2018), technology-related literature published in peer-reviewed journals. The PubMed search was conducted by querying the database with the device name (e.g., “Rapid AI”), and company websites were searched for “research” or “data” pages that cited studies involving the company’s technology. Studies published before 2018 were excluded, and we did not include review articles, editorials, or letters to the editor. Original, primary research that directly assessed the clinical performance of an AI-enabled technology for stroke diagnosis or management in humans was included. For each publication, we collected data regarding metrics commonly used to assess ML algorithm performance, such as accuracy, specificity, sensitivity, positive predictive value, negative predictive value, and area under the receiver-operator curve (AUC). We report results from 45 publications. This study was IRB-exempt.

3. Review of Literature

Our search revealed 22 FDA-approved, AI/ML-enabled technologies indicated for stroke diagnosis and management. A total of 18 companies developed these 22 technologies, with a majority (11/18; 61%) headquartered outside of the United States. The first approval was in February 2018. All initial approvals were for technologies that assist with ischemic stroke diagnosis, but two out of the last three approvals (BrainQ and IpsiHand) were for devices indicated for post-stroke rehabilitation (Table 1). Here, we synthesize the most recent literature on the clinical performance of these technologies.

4. Large Vessel Occlusion (LVO) Identification in Acute Ischemic Stroke

An important application of AI/ML is the automated detection of large vessel occlusions. Viz ContaCT, commercially known as Viz LVO, was the first FDA-approved AI/ML-enabled technology indicated for stroke and uses a convolutional neural network (CNN) as the underlying algorithm to detect LVOs from CT angiography (CTA). In data submitted to the FDA, Viz LVO displayed an area under the receiver operating curve (AUC) of 0.91 and reduced time from scan reading to specialist notification from 58 to 7 min [20], indicating improvement of clinical workflow efficiency. Others found similar increases in efficiency when using Viz LVO, reporting decreased transfer and stroke team notification times [21,22], as well as lengths of stay in the neurological ICU [22] (Table 2). Assessment of Viz LVO’s performance has shown negative predictive values (NPV) ranging from 79 to 99% and sensitivities between 81 and 88%, with relatively fast run times (~3 min) and consistent performance across different vascular structures [23,24] (Figure 2A). Notably, Viz LVO is an application within the broader Viz.ai platform, which includes tissue perfusion analysis on CTP and ICH identification on CT of the head.
RapidAI is a technology platform similar to Viz.ai. In addition to LVO identification on CTA (RAPID-CTA, RAPID-LVO), RapidAI includes software to analyze CT perfusion (RAPID-CTP) and MRI (RAPID-MRI) images for stroke triaging [36]. Though RAPID-LVO has a reported NPV range of 97–99% [26] and sensitivity ranging from 80–94%, there is a wide range of reported positive predictive values (PPV). Importantly, the PPV is 14% when identifying LVOs in the M2 segment of the MCA [26]. This is in contrast to Viz LVO’s reported lower bound PPV of 65%, which did not vary significantly across ICA, M1-MCA, and M2-MCA [24]. Variations in and relatively low PPVs highlight the use of these platforms as initial screening tools (given their high sensitivities and negative predictive values) that require subsequent expert confirmation to determine the presence of LVO (Table 2, Figure 2A). Use of both RAPID and Viz LVO has improved clinical workflows/outcomes (e.g., reducing CT to groin puncture times) with similar run times of ~3 min per scan [21,22,25].
Newer technologies for LVO identification include CINA-LVO [37] and HALO [38], which have shown promising performance in the few studies that have assessed their functionality. CINA has demonstrated relatively strong performance (PPV of 86–99%, NPV of 64–99%) across LVO anatomy [27,28]. The limited data for HALO reports an NPV of 91% and a PPV of 47%; however, performance varied based on the anatomical location of the LVO, with the lowest performance in M2 LVOs [29].

5. CT Head (CTH) Analysis (ASPECTS Score) in Acute Ischemic Stroke

Assessing the extent of irreversible ischemic damage to guide treatment decisions is equally important as identifying suspected LVOs. The Alberta Stroke Program Early CT Score (ASPECTS) is one widely used method for accomplishing this task. While diffusion-weighted MR imaging provides the most accurate information regarding acute infarction, CTH is more readily available in the acute setting. FDA-approved Rapid ASPECTS determines ASPECTS from CTs in patients with known MCA or ICA occlusions, but not for primary interpretation of CT images. In addition, the technology is only intended for use on GE Lightspeed VCT Scanners [39]. Overall, many have shown a strong correlation between ASPECTS determined manually by experts (e.g., neuroradiologists), which is currently the gold standard, and those calculated by Rapid ASPECTS [30,31,35] (Table 2). Some even report superior performance by Rapid ASPECTS in analyzing imaging obtained soon after symptom onset [32,34]. Rapid ASPECTS’ individual impact on clinical efficiency and patient outcomes has not yet been studied. However, use of the broader RapidAI mobile app, which includes Rapid ASPECTS functionality, decreased door-to-groin puncture times and improved subsequent NIH stroke scale scores [33].

6. CT Perfusion (CTP) Analysis in Acute Ischemic Stroke

Another class of FDA-approved, AI/ML-enabled technologies for the management of stroke includes technologies that analyze CTP or MR perfusion images to assess the core and penumbra volumes and predict final infarct volumes. CTP can demonstrate ischemic tissue, which consists of non-salvageable tissue and at-risk tissue that could be rescued with successful reperfusion. CTP analysis provides specific parameters, including cerebral blood volume (CBV), cerebral blood flow (CBF), and mean transit time (MTT). Rapid-CTP is a comprehensively studied tool for CTP analysis within the broader RAPID platform and performs well in estimating final infarct volumes, with high accuracy and relatively strong correlations to the gold standard (e.g., human estimates of volumes) [40,41,42,43,44,45] (Table 3). Vitrea CT Brain Perfusion was approved by the FDA in November 2018 to quantify cerebral blood flow and predict final infarct volumes [46]. Many groups have found Vitrea outperforms Rapid-CTP with respect to final infarct volume predictions [47,48,49], with the gold standard determined by human interpretation of DWI/FLAIR imaging (Table 3; Figure 2B). FastStroke/CT Perfusion 4D is a similar technology that not only predicts ischemic core volume but also assesses the quantity and quality of collateral perfusion [50,51]. Similar to Vitrea CT, FastStroke/CT Perfusion 4D performed comparably to Rapid-CTP (intraclass correlation coefficient of 0.95) [52], and its additional capability to assess collateral circulation improved accuracy in predicting good outcomes [53]. Icobrain CTP uses a CNN to estimate penumbra volumes and cerebral blood flow, both of which have strong correlations to expert assessments by radiologists [54,55] (Table 3). Viz CTP is a similar software that performed well in predicting final infarct volume (r = ~0.6) [56]. While the above software solutions are well-characterized, there are no studies demonstrating improved time-to-reperfusion. Solutions such as Augmented Vascular Analysis [57] and Neuro.AI Algorithm [58] are yet to be independently assessed in the literature (Table 3).

7. Intracranial Hemorrhage (ICH) Identification

Technologies indicated for the detection of ICH generally performed better than those indicated for LVO detection (Table 4). BriefCase was the first FDA-approved, AI/ML-enabled technology for the identification of ICH from non-contrast head CT [62]. BriefCase’s CNN-based algorithm [63] has shown strong performance by reducing outpatient scan interpretation delays by 90% (604 min reduction) and inpatient delays by 10% (38 min reduction) [64]. Cases flagged by BriefCase as suspicious for ICH had an average turnaround time of 73 min, versus 132 min for non-flagged cases [65]. Recent studies assessing BriefCase have reported NPVs of 96–99% and PPVs of 72–96% [64,66,67] (Figure 2C). A main driver of false negatives was ICH anatomy (e.g., under the calvaria), while false positives were driven by tumors and calcifications [68,69].
CINA-ICH has similar reported performance in ICH detection compared to BriefCase (Figure 2C). NPVs ranged from 92–99%, PPVs from 80–97%, and the algorithm had a sensitivity of 72% when identifying relatively small-volume bleeds (volume less than 5 mL) [27,70]. CINA has additional subclassification functionality (e.g., differentiating between subarachnoid and intraventricular hemorrhage) with a sensitivity of at least 90% [27]. CuraRad-ICH, on the other hand, had subclassification sensitivities between 61 and 99% [71,72], though the software was studied on a larger sample of scans and has specificities roughly comparable to those of CINA.
Rapid-ICH [73], with PPV, NPV, accuracy, sensitivity, and specificity of at least 95% [74], and HealthICH [75], with an AUC of 0.96 [76], are two other technologies indicated for ICH detection. Some FDA-approved technologies for ICH detection have yet to be studied independently in the literature. These include Accipiolx [77], DeepCT [78], NinesAI [79], qER [80], and Viz ICH [81].
Table
Table 4. Ten technologies indicated to diagnose ICH. The majority of studies were retrospective, and algorithm performance met or exceeded human performance in binary and multiclass classification. ICH location (e.g., under the calvaria) and anatomical variations (e.g., calcification of the falx) reduced algorithm performance. Human performance generally continues to be the gold standard for evaluating these algorithms. CPH: Cerebral Parenchymal Hemorrhage; EDH: Extradural hemorrhage; ICA: Internal Carotid Artery; IVH: Intraventricular Hemorrhage; MCA: Middle Cerebral Artery; SAH: Subarachnoid Hemorrhage. * Indicates metric was extrapolated from available data.
Table 4. Ten technologies indicated to diagnose ICH. The majority of studies were retrospective, and algorithm performance met or exceeded human performance in binary and multiclass classification. ICH location (e.g., under the calvaria) and anatomical variations (e.g., calcification of the falx) reduced algorithm performance. Human performance generally continues to be the gold standard for evaluating these algorithms. CPH: Cerebral Parenchymal Hemorrhage; EDH: Extradural hemorrhage; ICA: Internal Carotid Artery; IVH: Intraventricular Hemorrhage; MCA: Middle Cerebral Artery; SAH: Subarachnoid Hemorrhage. * Indicates metric was extrapolated from available data.
DeviceAuthor, YearLevel of EvidenceDataset CharacteristicsSample Size (Scans)AUCPPVNPVAccuracySensitivitySpecificityOther Metrics/Comments
BriefCaseOjeda et al., 2019 [63]RetrospectiveProprietary,
Multicenter		7112-96%98%98%95%99%BriefCase uses a CNN to analyze non-contrast CTs to detect and triage ICH.
Wismüller et al., 2020 [65]Randomized Clinical TrialProprietary,
Single Center		620---96%95%97%Turn-around times for cases flagged by BriefCase (73 min) were significantly lower than those for non-flagged cases (132 min).
Ginat et al., 2020 [66]ProspectiveProprietary,
Single Center		2011-74%98%93%89%94%Accuracy was significantly higher for emergency (96.5%) vs. inpatient (89.4%) cases. False positives had various causes, including: (1) artifacts, (2) thick dura, (3) intra-arterial clot, (4) calcifications, and (5) tumors.
Rao et al., 2021 [69]RetrospectiveProprietary,
Single Center		5585------When applied to scans that radiologists reported as negative for ICH, BriefCase found 28 scans with ICH, of which 16 truly did. Subset analysis showed a false positive rate of 32%.
Ginat et al., 2021 [64]RetrospectiveProprietary,
Single Center		8723-86%96%-88%96%Scan view delay for cases flagged by the software decreased by 37 min for inpatients and 604 min for outpatients. In the ER, time reduction was most prominent during the 9 p.m. to 3 a.m. and 10 a.m. to 12 p.m. periods, and especially during the weekend.
Voter et al., 2021 [67]RetrospectiveProprietary,
Single Center		3605-81%99%96% *92%98%Neuroradiologists and the software agreed 97% of the time. Prior neurosurgery decreased model performance.
Kundisch et al., 2021 [68]RetrospectiveProprietary,
Multicenter		4946-72% *99% *97% *88% *98% *Software detected 29 additional ICHs (0.59%) in the cohort. False negative rate was 12.4% compared to the radiologist rate of 10.9%. Anatomical variations (e.g., calcifications) were difficult for the algorithm to analyze.
CINAMcLouth et al., 2021 [27]RetrospectiveProprietary,
Multicenter		814-80–97%92–99%96%91%97%True positive rates (sensitivity) for ICH subclassification were >90%. ICH < 5 mL had a sensitivity of 72%.
Rava et al., 2021 [70]RetrospectiveProprietary,
Single Center		302-85%98%94%93%93%95% of ICH volumes were correctly triaged. 88% of non-ICH cases were correctly classified as ICH negative.
CuraRad-ICHYe et al., 2019 [71]RetrospectiveProprietary,
Multicenter		28360.8–1.0--75–99%61–99%82–99%Algorithm was evaluated for binary classification (ICH vs. no ICH) and multi-type classification (CPH, SAH, EDH, SDH, IVH).
Guo et al., 2020 [72]RetrospectiveProprietary,
Multicenter		11760.85–0.99--90–98%78–97%92–100%Algorithm was evaluated for binary classification (ICH vs. no ICH) and multi-type classification (CPH, SAH, EDH, SDH, IVH).
Rapid ICHHeit et al., 2021 [74]RetrospectiveProprietary,
Multicenter		308-96%95%95% *96%95%
HealthICHBar et al., 2018 [76]RetrospectiveProprietary,
Multicenter		14260.96-----
Accipiolx
DeepCT
NinesAI
QER
Viz ICH

8. Rehabilitation

Tools for post-stroke rehabilitation require further development, especially given the poor natural recovery that is often seen with stroke [62]. There is a need for technologies that can extend the therapeutic window for patients and/or enable neurological recovery. Here, we describe two FDA-approved technologies that can enhance post-stroke recovery.
An Israeli-based company, BrainQ, is developing a non-invasive brain-computer interface (BCI) device that leverages extremely low frequency and low intensity electromagnetic fields (ELF-EMF) to promote post-stroke recovery [82,83]. After a stroke, patients often have abnormal neural oscillatory patterns, and exposure to tuned EMFs can influence these oscillations [84], thereby promoting periods of neuroplasticity [85,86]. BrainQ’s technology uses ML to extract motor-related spectral features from electrophysiology measurements (EEG, MEG/EMG) [87] and then translates these into a specific ELF-EMF treatment for patients [88].
BrainQ received FDA breakthrough status in February 2021 based on results from a pilot trial of 25 patients with a history of sub-acute ischemic stroke. Patients who received 40 min of ELF-EMF treatment 5 days a week for 8 weeks had superior recovery compared to the sham group as assessed by multiple metrics (e.g., NIH stroke score) and did not report any adverse events [89]. BrainQ has planned a double-blind national clinical trial across up to 20 inpatient rehabilitation facilities in the United States [90]. A previous BrainQ clinical trial was terminated due to the COVID-19 pandemic [91].
IpsiHand Upper Extremity Rehabilitation System (IpsiHand), granted breakthrough status by the FDA in April 2021, is the first FDA-approved device to use BCI technology to facilitate motor rehabilitation in patients who are more than 6 months post-stroke. The device uses an EEG electrode headset to translate neural activity of movement intent from the uninjured brain hemisphere into physical movements of a robotic exoskeleton worn around the impaired hand, wrist, and forearm [92]. A study of ten chronic hemiparetic stroke survivors with upper-limb impairment showed significant improvement in arm functionality after 12 weeks of IpsiHand therapy, with only minor side effects (e.g., skin redness) [93]. A randomized clinical trial is needed to assess whether use of IpsiHand alone proves more beneficial for upper extremity function versus traditional physical therapy. IpsiHand has the potential to enhance functional recovery with convenient, in-home post-stroke rehabilitation.

9. Discussion

In total, the FDA has approved 22 unique AI/ML-enabled technologies to assist clinicians with the diagnosis or management of stroke or ICH. The 20 technologies indicated for assistance with diagnosis can save valuable time by triaging potentially troubling scans and reducing the need for labor-intensive and time-consuming tasks such as segmentation. These technologies can ultimately reduce delays for patients to receive life-saving interventions. Adoption of these technologies has been strong, with RAPID in use at 1800 and Viz.ai in 900 hospitals [94,95]. In addition, some of the technologies described here can co-function within a broader technology suite to facilitate care coordination. For example, technologies developed by the same company (e.g., Viz.ai, RapidAI) are hosted within an interconnected system that includes mobile alerts, capabilities for remote CT/MRI viewing, and HIPAA-compliant provider-to-provider communication. This simplification of care coordination works synergistically with AI/ML capabilities to achieve the results these technologies have produced. In the future, more of these technologies will need to be housed within similar, integrated clinical systems. Such technology-based care coordination and workflow simplification solutions have improved outcomes in non-stroke medical emergency settings [96]. The two devices indicated for post-stroke rehabilitation are leveraging AI/ML to create new forms of therapies that are opening the possibility for patients to achieve significant recovery following serious neurological injury. With time, the safety and capabilities of these devices will only improve, further enabling clinicians to facilitate favorable patient outcomes. In the future, there will be a need for head-to-head comparisons of technologies using the same clinical datasets to further help clinicians and health systems more definitively decide which to use.
Given the proprietary nature of the technologies discussed here, specific details about ML model design and algorithms are not publicly available. However, most models that underlie technologies indicated for stroke diagnosis leverage deep learning in the form of convolutional neural networks. The reasons for performance variation across these technologies are multifactorial and likely due to a combination of model design and quality/quantity of testing data. Model design in ML development involves determining model structure (e.g., number of layers in a neural network) and hyperparameters (e.g., learning rate for optimization algorithms) and still relies on trial and error [97,98,99,100]. When testing different model designs, developers must also keep end-user performance and experience in mind, as these technologies must work across multiple imaging platforms and user devices. Differences in the quality and quantity of training data also likely contribute to differences in performance among the technologies discussed here. Currently, the “gold-standard” used to train and measure the success metrics of the models underlying these technologies is produced by radiologists and other clinicians. Therefore, any inherent human error in this data will be carried forward and learned by the algorithm. This is primarily an issue for algorithms that require segmentation for tasks such as CT perfusion analysis and ICH detection. Classification algorithms (e.g., triaging CT scans for stroke) are less impacted by this human error, but their performance is influenced by the specific training methods (e.g., cross-validation, dropout regularization) that were employed during algorithm development. Finally, the size and variety of training data play a large role in algorithm performance. Training a model with a large number of unique data points is important to minimize overfitting and thereby maximize performance. The majority of the publications analyzed here used proprietary, single-center datasets to evaluate technologies (Table 2, Table 3 and Table 4) and there is a need for larger, multi-center, international research in order to more comprehensively test these technologies across a variety of clinical scenarios and patient populations.
There is great promise for AI/ML-enabled technologies in stroke diagnosis and management. The automated process can minimize intra-/inter-rater variability and provide support for less experienced or non-specialized physicians. As such, integration of these software solutions into patient care can improve the speed and accuracy of diagnosis. However, there are still issues to be addressed. The first is improving algorithm performance. Though many studies have shown that these AI/ML algorithms can perform comparably to or even outperform neuroradiologists, the technologies can fall short in certain instances. The anatomical location of the disease or patient-to-patient anatomical variation is a common cause of impaired algorithm performance. Difficulties with LVO identification were seen with anatomical variations such as early unilateral MCA or petrous ICA [26]. Additionally, LVO identification is less reliable for the posterior circulation, and algorithms encounter difficulty determining distal versus proximal occlusions. Similarly, AI-enabled software has had difficulty identifying ICH immediately under the calvaria [68], likely driven by a combination of beam-hardening and partial volume artifacts in CT imaging [101]. Anatomical variation, such as calcification of the falx cerebri, can also make ICH identification difficult [69]. Continued training of algorithms in patients with anatomical variations or additional intracranial abnormalities will be needed to improve performance. Fortunately, training functionalities can be developed that enable clinicians to inform the algorithm when errors are made. As discussed above, there is still a lack of robust, head-to-head comparisons between these various technologies, making it difficult to clearly identify one as “superior” to others. The technologies discussed here can safely be used in clinical practice to augment the expertise of radiologists, neurologists, and neurosurgeons, keeping in mind the performance data and shortcomings presented here.
The second issue to address with the use of these technologies involves the use of black-box AI/ML algorithms. Neural networks, and “deep” neural networks in particular, can tend to be “black box” algorithms in that they arrive at conclusions and outputs without readily providing information regarding their analytical process. As neural networks tend to underlie many AI/ML-enabled tools in healthcare today, some have argued that these black-box algorithms must have increased transparency before being used in high-stakes healthcare settings. There have been efforts to make AI/ML algorithms more interpretable to the end user [102,103], with current “explainability methods” satisfying desires for transparency to various extents [104]. Two common methods include saliency maps, which visually highlight key features that an algorithm used to make a decision, and feature relevance, which is a list of key quantitative or qualitative features that an algorithm used in decision-making [105]. The field should also focus on rigorous internal and external validation of algorithms and compliance with a common set of machine learning best practices that support a level of standardization within the field. The FDA’s plan to develop new protocols for assessing AI/ML-enabled technologies also addresses these concerns [6]. The role of intellectual property is a balancing concern that must be acknowledged. Though it would be beneficial for end users to understand how exactly a decision was made to, for example, not triage a scan for possible LVO, technology developers may worry that publicly releasing too much information could potentially compromise security of intellectual property. Still, there must be a common ground on which adequate information is given to users or regulators about how, for example, a piece of software makes decisions when analyzing radiology images.
The last issue that must be addressed involves reimbursement for hospitals that utilize these technologies. In 2018, the Centers for Medicare and Medicaid Services (CMS) approved payment for use of Viz LVO through a “New Technology Add-on Payment”, marking the first time CMS reimbursed AI/ML-enabled software through this mechanism [106]. RapidAI also received this designation from CMS [107], marking important administrative milestones to increase the use of these technologies in direct patient care. There is little public data available regarding the pricing of these technologies, and specific pricing structures likely vary between contracts across different hospitals. In the future, health systems must be aware of the reimbursement they receive when using these novel technologies when negotiating licensing contracts. Additional potential costs to a health system that must also be considered include increased expenditure on computing power and physical or cloud storage.
Overall, the studies analyzed here show that AI/ML-enabled technologies for stroke diagnosis are performing equally to or even exceeding human performance on certain metrics. Generally, these technologies have higher sensitivities and NPVs compared to specificities and PPVs, further highlighting their role in triaging and assisting clinicians rather than making definitive clinical decisions. In the coming years, as these technologies are exposed to more cases and increased clinical data, their performance should only improve, but as stated above, there is still a need for larger, multi-center trials to continue monitoring and comparing technology performance. There is a common theme in the studies assessed here: the use of these technologies can improve clinical workflows. However, few studies further correlated increased efficiency with quantifiable improvements in patient outcomes. Some studies report improvements in, for example, the NIH stroke scale, but there is still perhaps a need for long-term follow-up to assess if the workflow efficiency facilitated by these technologies truly translates to better long-term clinical outcomes. Similarly, there are inconsistencies across studies regarding the timing of imaging analysis, as only a few assessed the differential impact of using AI/ML-enabled technologies in the acute (e.g., 0–6 h) versus the sub-acute (e.g., 6–24 h) timeframes. This is particularly important when analyzing zones of ischemia and the viability of tissue post-stroke. In the same vein, there must be more consistency in reporting the time technologies take to evaluate imaging. As the complexity of medical imaging increases, the algorithms underlying these technologies must be refined to efficiently process scans at a speed that is at least equivalent to, if not faster than, human performance.
To facilitate the approval and clinical utilization of emerging AI/ML-enabled technologies, the FDA has created protocols to better assist researchers in developing these technologies and navigating the FDA approval process [6]. These protocols outlined good machine learning practices for researchers to follow, created guidelines for algorithm transparency, and established more robust guidelines for real-world data collection. As the field of AI/ML rapidly advances, these protocols must also undergo rapid revision to remain relevant. This includes new guidelines related to data security, bias in data sources, and post-approval monitoring. In the future, when more technologies are developed not only for stroke diagnosis but also for stroke prediction in high-risk populations and chronic stroke management, robust guidelines will be essential as these technologies are used in larger patient populations.
Our study has a few limitations. First, though the majority of papers analyzed here were identified in PubMed, some papers were only discovered after searching company websites. This may have introduced selection bias to our literature search, though the minority of papers were found by searching company websites. Second, our study focuses on FDA-approved solutions, but there are many emerging technologies in the research pipeline that are not discussed here. Third, given the proprietary nature of the technologies described here, we are unable to specifically comment on the strengths and weaknesses of the design of the algorithms that underlie these technologies.

10. Conclusions

FDA-approved AI/ML-enabled technologies for stroke diagnosis and management have proven to be powerful tools in improving the efficiency and accuracy of patient care decisions by physicians. Many of these technologies use convolutional neural networks as their underlying algorithm and have approached or even exceeded the gold standard of human performance when tested using real-world data. Future work is needed to further refine the performance of these technologies in, for example, patients with aberrant anatomy and to compare technologies in a head-to-head manner with large, multi-center studies.

Author Contributions

Conceptualization, A.S.C., E.A.K. and P.M.; methodology, A.S.C., E.A.K., M.P. and P.M.; formal analysis, A.S.C. and E.A.K.; data curation, A.S.C. and E.A.K.; writing—original draft preparation, A.S.C., E.A.K., J.D.S., R.T.K. and P.M.; writing—review and editing, A.S.C., E.A.K., J.D.S., R.T.K., M.P. and P.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

IRB approval was not required for this study.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data used in this study are publicly available to researchers through the U.S. National Library of Medicine. Additional inquiries are welcome to the corresponding author.

Conflicts of Interest

Ryan T. Kellogg is a consultant for Viz.ai.

References

  1. Virani, S.S.; Alonso, A.; Benjamin, E.J.; Bittencourt, M.S.; Callaway, C.W.; Carson, A.P.; Chamberlain, A.M.; Chang, A.R.; Cheng, S.; Delling, F.N.; et al. Heart Disease and Stroke Statistics—2020 Update: A Report From the American Heart Association. Circulation 2020, 141, e139–e596. [Google Scholar] [CrossRef] [PubMed]
  2. An, S.J.; Kim, T.J.; Yoon, B.-W. Epidemiology, Risk Factors, and Clinical Features of Intracerebral Hemorrhage: An Update. J. Stroke 2017, 19, 3–10. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Mendelson, S.J.; Prabhakaran, S. Diagnosis and Management of Transient Ischemic Attack and Acute Ischemic Stroke: A Review. JAMA 2021, 325, 1088–1098. [Google Scholar] [CrossRef] [PubMed]
  4. Meretoja, A.; Keshtkaran, M.; Tatlisumak, T.; Donnan, G.A.; Churilov, L. Endovascular Therapy for Ischemic Stroke: Save a Minute-Save a Week. Neurology 2017, 88, 2123–2127. [Google Scholar] [CrossRef]
  5. Bohr, A.; Memarzadeh, K. The Rise of Artificial Intelligence in Healthcare Applications. Artif. Intell. Healthc. 2020, 25–60. [Google Scholar] [CrossRef]
  6. Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. 2021. Available online: https://www.fda.gov/media/145022/download (accessed on 18 October 2021).
  7. Sarvestany, S.S.; Kwong, J.C.; Azhie, A.; Dong, V.; Cerocchi, O.; Ali, A.F.; Karnam, R.S.; Kuriry, H.; Shengir, M.; Candido, E.; et al. Development and Validation of an Ensemble Machine Learning Framework for Detection of All-Cause Advanced Hepatic Fibrosis: A Retrospective Cohort Study. Lancet Digit. Health 2022, 4, e188–e199. [Google Scholar] [CrossRef]
  8. Bos, J.M.; Attia, Z.I.; Albert, D.E.; Noseworthy, P.A.; Friedman, P.A.; Ackerman, M.J. Use of Artificial Intelligence and Deep Neural Networks in Evaluation of Patients With Electrocardiographically Concealed Long QT Syndrome From the Surface 12-Lead Electrocardiogram. JAMA Cardiol. 2021, 6, 532–538. [Google Scholar] [CrossRef]
  9. Chandrabhatla, A.S.; Pomeraniec, I.J.; Ksendzovsky, A. Co-Evolution of Machine Learning and Digital Technologies to Improve Monitoring of Parkinson’s Disease Motor Symptoms. Npj Digit. Med. 2022, 5, 1–18. [Google Scholar] [CrossRef]
  10. Thompson, A.C.; Jammal, A.A.; Berchuck, S.I.; Mariottoni, E.B.; Medeiros, F.A. Assessment of a Segmentation-Free Deep Learning Algorithm for Diagnosing Glaucoma From Optical Coherence Tomography Scans. JAMA Ophthalmol. 2020, 138, 333–339. [Google Scholar] [CrossRef]
  11. Le Page, A.L.; Ballot, E.; Truntzer, C.; Derangère, V.; Ilie, A.; Rageot, D.; Bibeau, F.; Ghiringhelli, F. Using a Convolutional Neural Network for Classification of Squamous and Non-Squamous Non-Small Cell Lung Cancer Based on Diagnostic Histopathology HES Images. Sci. Rep. 2021, 11, 23912. [Google Scholar] [CrossRef]
  12. Sirsat, M.S.; Fermé, E.; Câmara, J. Machine Learning for Brain Stroke: A Review. J. Stroke Cereb. Dis. 2020, 29, 105162. [Google Scholar] [CrossRef] [PubMed]
  13. Mainali, S.; Darsie, M.E.; Smetana, K.S. Machine Learning in Action: Stroke Diagnosis and Outcome Prediction. Front. Neurol. 2021, 12, 734345. [Google Scholar] [CrossRef] [PubMed]
  14. Chavva, I.R.; Crawford, A.L.; Mazurek, M.H.; Yuen, M.M.; Prabhat, A.M.; Payabvash, S.; Sze, G.; Falcone, G.J.; Matouk, C.C.; de Havenon, A.; et al. Deep Learning Applications for Acute Stroke Management. Ann. Neurol. 2022, 92, 574–587. [Google Scholar] [CrossRef] [PubMed]
  15. Shlobin, N.A.; Baig, A.A.; Waqas, M.; Patel, T.R.; Dossani, R.H.; Wilson, M.; Cappuzzo, J.M.; Siddiqui, A.H.; Tutino, V.M.; Levy, E.I. Artificial Intelligence for Large-Vessel Occlusion Stroke: A Systematic Review. World Neurosurg. 2022, 159, 207–220.e1. [Google Scholar] [CrossRef] [PubMed]
  16. Campagnini, S.; Arienti, C.; Patrini, M.; Liuzzi, P.; Mannini, A.; Carrozza, M.C. Machine Learning Methods for Functional Recovery Prediction and Prognosis in Post-Stroke Rehabilitation: A Systematic Review. J. Neuroeng. Rehabil. 2022, 19, 54. [Google Scholar] [CrossRef]
  17. Zhu, S.; Gilbert, M.; Chetty, I.; Siddiqui, F. The 2021 Landscape of FDA-Approved Artificial Intelligence/Machine Learning-Enabled Medical Devices: An Analysis of the Characteristics and Intended Use. Int. J. Med. Inform. 2022, 165, 104828. [Google Scholar] [CrossRef]
  18. Benjamens, S.; Dhunnoo, P.; Meskó, B. The State of Artificial Intelligence-Based FDA-Approved Medical Devices and Algorithms: An Online Database. Npj Digit. Med. 2020, 3, 1–8. [Google Scholar] [CrossRef]
  19. Lyell, D.; Coiera, E.; Chen, J.; Shah, P.; Magrabi, F. How Machine Learning Is Embedded to Support Clinician Decision Making: An Analysis of FDA-Approved Medical Devices. BMJ Health Care Inf. 2021, 28, e100301. [Google Scholar] [CrossRef]
  20. US Food and Drug Administration. Viz ContaCT/LVO 513(f)(2) de Novo Letter (DEN170073). Available online: https://www.accessdata.fda.gov/cdrh_docs/reviews/DEN170073.pdf (accessed on 25 December 2021).
  21. Morey, J.R.; Zhang, X.; Yaeger, K.A.; Fiano, E.; Marayati, N.F.; Kellner, C.P.; De Leacy, R.A.; Doshi, A.; Tuhrim, S.; Fifi, J.T. Real-World Experience with Artificial Intelligence-Based Triage in Transferred Large Vessel Occlusion Stroke Patients. Cereb. Dis 2021, 50, 450–455. [Google Scholar] [CrossRef]
  22. Hassan, A.E.; Ringheanu, V.M.; Rabah, R.R.; Preston, L.; Tekle, W.G.; Qureshi, A.I. Early Experience Utilizing Artificial Intelligence Shows Significant Reduction in Transfer Times and Length of Stay in a Hub and Spoke Model. Interv. Neuroradiol. 2020, 26, 615–622. [Google Scholar] [CrossRef]
  23. Yahav-Dovrat, A.; Saban, M.; Merhav, G.; Lankri, I.; Abergel, E.; Eran, A.; Tanne, D.; Nogueira, R.G.; Sivan-Hoffmann, R. Evaluation of Artificial Intelligence-Powered Identification of Large-Vessel Occlusions in a Comprehensive Stroke Center. AJNR Am. J. Neuroradiol. 2021, 42, 247–254. [Google Scholar] [CrossRef] [PubMed]
  24. Rodrigues, G.; Barreira, C.M.; Bouslama, M.; Haussen, D.C.; Al-Bayati, A.; Pisani, L.; Liberato, B.; Bhatt, N.; Frankel, M.R.; Nogueira, R.G. Automated Large Artery Occlusion Detection in Stroke: A Single-Center Validation Study of an Artificial Intelligence Algorithm. Cereb. Dis. 2022, 51, 259–264. [Google Scholar] [CrossRef] [PubMed]
  25. Adhya, J.; Li, C.; Eisenmenger, L.; Cerejo, R.; Tayal, A.; Goldberg, M.; Chang, W. Positive Predictive Value and Stroke Workflow Outcomes Using Automated Vessel Density (RAPID-CTA) in Stroke Patients: One Year Experience. Neuroradiol. J. 2021, 34, 476–481. [Google Scholar] [CrossRef] [PubMed]
  26. Amukotuwa, S.A.; Straka, M.; Smith, H.; Chandra, R.V.; Dehkharghani, S.; Fischbein, N.J.; Bammer, R. Automated Detection of Intracranial Large Vessel Occlusions on Computed Tomography Angiography. Stroke 2019, 50, 2790–2798. [Google Scholar] [CrossRef] [PubMed]
  27. McLouth, J.; Elstrott, S.; Chaibi, Y.; Quenet, S.; Chang, P.D.; Chow, D.S.; Soun, J.E. Validation of a Deep Learning Tool in the Detection of Intracranial Hemorrhage and Large Vessel Occlusion. Front. Neurol. 2021, 12, 655. [Google Scholar] [CrossRef]
  28. Rava, R.A.; Peterson, B.A.; Seymour, S.E.; Snyder, K.V.; Mokin, M.; Waqas, M.; Hoi, Y.; Davies, J.M.; Levy, E.I.; Siddiqui, A.H.; et al. Validation of an Artificial Intelligence-Driven Large Vessel Occlusion Detection Algorithm for Acute Ischemic Stroke Patients. Neuroradiol. J. 2021, 34, 408–417. [Google Scholar] [CrossRef] [PubMed]
  29. Luijten, S.P.R.; Wolff, L.; Duvekot, M.H.C.; van Doormaal, P.-J.; Moudrous, W.; Kerkhoff, H.; Lycklama A Nijeholt, G.J.; Bokkers, R.P.H.; Yo, L.S.F.; Hofmeijer, J.; et al. Diagnostic Performance of an Algorithm for Automated Large Vessel Occlusion Detection on CT Angiography. J. Neurointerv. Surg. 2022, 14, 794–798. [Google Scholar] [CrossRef] [PubMed]
  30. Lasocha, B.; Pulyk, R.; Brzegowy, P.; Latacz, P.; Slowik, A.; Popiela, T.J. Real-World Comparison of Human and Software Image Assessment in Acute Ischemic Stroke Patients’ Qualification for Reperfusion Treatment. J. Clin. Med. 2020, 9, 3383. [Google Scholar] [CrossRef]
  31. Hoelter, P.; Muehlen, I.; Goelitz, P.; Beuscher, V.; Schwab, S.; Doerfler, A. Automated ASPECT Scoring in Acute Ischemic Stroke: Comparison of Three Software Tools. Neuroradiology 2020, 62, 1231–1238. [Google Scholar] [CrossRef]
  32. Maegerlein, C.; Fischer, J.; Mönch, S.; Berndt, M.; Wunderlich, S.; Seifert, C.L.; Lehm, M.; Boeckh-Behrens, T.; Zimmer, C.; Friedrich, B. Automated Calculation of the Alberta Stroke Program Early CT Score: Feasibility and Reliability. Radiology 2019, 291, 141–148. [Google Scholar] [CrossRef]
  33. Al-Kawaz, M.; Primiani, C.; Urrutia, V.; Hui, F. Impact of RapidAI Mobile Application on Treatment Times in Patients with Large Vessel Occlusion. J. NeuroInterventional Surg. 2021, 14, 233–236. [Google Scholar] [CrossRef] [PubMed]
  34. Albers, G.W.; Wald, M.J.; Mlynash, M.; Endres, J.; Bammer, R.; Straka, M.; Maier, A.; Hinson, H.E.; Sheth, K.N.; Taylor Kimberly, W.; et al. Automated Calculation of Alberta Stroke Program Early CT Score: Validation in Patients with Large Hemispheric Infarct. Stroke 2019, 50, 3277–3279. [Google Scholar] [CrossRef] [PubMed]
  35. Mansour, O.Y.; Ramadan, I.; Abdo, A.; Hamdi, M.; Eldeeb, H.; Marouf, H.; Elsalamawy, D.; Elfatatry, A.; Elnekidy, A.; Reda, M.I. Deciding Thrombolysis in AIS Based on Automated versus on WhatsApp Interpreted ASPECTS, a Reliability and Cost-Effectiveness Analysis in Developing System of Care. Front. Neurol. 2020, 11, 333. [Google Scholar] [CrossRef]
  36. Robert Ochs US Food and Drug Administration, Division of Radiological Health. ISchemaView RAPID 510(k) Premarket Notification Letter (K182130). 2018. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf18/K182130.pdf (accessed on 24 December 2021).
  37. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. CINA 510(k) Premarket Notification Letter (K200855). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf20/K200855.pdf (accessed on 21 December 2021).
  38. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. HALO 510(k) Premarket Notification Letter (K200873). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf20/K200873.pdf (accessed on 27 December 2021).
  39. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. Rapid ASPECTS 510(k) Premarket Notification Letter (K200760). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf20/K200760.pdf (accessed on 27 December 2021).
  40. Hokkinen, L.; Mäkelä, T.; Savolainen, S.; Kangasniemi, M. Computed Tomography Angiography-Based Deep Learning Method for Treatment Selection and Infarct Volume Prediction in Anterior Cerebral Circulation Large Vessel Occlusion. Acta Radiol Open 2021, 10, 20584601211060348. [Google Scholar] [CrossRef] [PubMed]
  41. Hokkinen, L.; Mäkelä, T.; Savolainen, S.; Kangasniemi, M. Evaluation of a CTA-Based Convolutional Neural Network for Infarct Volume Prediction in Anterior Cerebral Circulation Ischaemic Stroke. Eur. Radiol. Exp. 2021, 5, 25. [Google Scholar] [CrossRef]
  42. Wouters, A.; Robben, D.; Christensen, S.; Marquering, H.A.; Roos, Y.B.W.E.M.; van Oostenbrugge, R.J.; van Zwam, W.H.; Dippel, D.W.J.; Majoie, C.B.L.M.; Schonewille, W.J.; et al. Prediction of Stroke Infarct Growth Rates by Baseline Perfusion Imaging. Stroke 2022, 53, 569–577. [Google Scholar] [CrossRef]
  43. Potreck, A.; Seker, F.; Mutke, M.A.; Weyland, C.S.; Herweh, C.; Heiland, S.; Bendszus, M.; Möhlenbruch, M. What Is the Impact of Head Movement on Automated CT Perfusion Mismatch Evaluation in Acute Ischemic Stroke? J. NeuroInterventional Surg. 2022, 14, 628–633. [Google Scholar] [CrossRef]
  44. Bouslama, M.; Ravindran, K.; Harston, G.; Rodrigues, G.M.; Pisani, L.; Haussen, D.C.; Frankel, M.R.; Nogueira, R.G. Noncontrast Computed Tomography E-Stroke Infarct Volume Is Similar to RAPID Computed Tomography Perfusion in Estimating Postreperfusion Infarct Volumes. Stroke 2021, 52, 634–641. [Google Scholar] [CrossRef]
  45. Kim, Y.-C.; Lee, J.-E.; Yu, I.; Song, H.-N.; Baek, I.-Y.; Seong, J.-K.; Jeong, H.-G.; Kim, B.J.; Nam, H.S.; Chung, J.-W.; et al. Evaluation of Diffusion Lesion Volume Measurements in Acute Ischemic Stroke Using Encoder-Decoder Convolutional Network. Stroke 2019, 50, 1444–1451. [Google Scholar] [CrossRef]
  46. Robert Ochs US Food and Drug Administration, Division of Radiological Health. VitreaCT Brain Perfusion 510(k) Premarket Notification Letter (K181247). 2018. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf18/K181247.pdf (accessed on 16 December 2021).
  47. Rava, R.A.; Snyder, K.V.; Mokin, M.; Waqas, M.; Allman, A.B.; Senko, J.L.; Podgorsak, A.R.; Shiraz Bhurwani, M.M.; Hoi, Y.; Siddiqui, A.H.; et al. Assessment of a Bayesian Vitrea CT Perfusion Analysis to Predict Final Infarct and Penumbra Volumes in Patients with Acute Ischemic Stroke: A Comparison with RAPID. AJNR Am. J. Neuroradiol. 2020, 41, 206–212. [Google Scholar] [CrossRef]
  48. Rava, R.A.; Snyder, K.V.; Mokin, M.; Waqas, M.; Allman, A.B.; Senko, J.L.; Podgorsak, A.R.; Bhurwani, M.M.S.; Davies, J.M.; Levy, E.I.; et al. Effect of Computed Tomography Perfusion Post-Processing Algorithms on Optimal Threshold Selection for Final Infarct Volume Prediction. Neuroradiol. J. 2020, 33, 273–285. [Google Scholar] [CrossRef]
  49. Rava, R.A.; Podgorsak, A.R.; Waqas, M.; Snyder, K.V.; Mokin, M.; Levy, E.I.; Davies, J.M.; Siddiqui, A.H.; Ionita, C.N. Investigation of Convolutional Neural Networks Using Multiple Computed Tomography Perfusion Maps to Identify Infarct Core in Acute Ischemic Stroke Patients. J. Med. Imaging 2021, 8, 014505. [Google Scholar] [CrossRef] [PubMed]
  50. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. FastStroke, CT Perfusion 4D 510(k) Premarket Notification Letter (K193289). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf19/K193289.pdf (accessed on 27 December 2021).
  51. Verdolotti, T.; Pilato, F.; Cottonaro, S.; Monelli, E.; Giordano, C.; Guadalupi, P.; Benenati, M.; Ramaglia, A.; Costantini, A.M.; Alexandre, A.; et al. ColorViz, a New and Rapid Tool for Assessing Collateral Circulation during Stroke. Brain Sci. 2020, 10, E882. [Google Scholar] [CrossRef] [PubMed]
  52. Liu, Q.C.; Jia, Z.Y.; Zhao, L.B.; Cao, Y.Z.; Ma, G.; Shi, H.B.; Liu, S. Agreement and Accuracy of Ischemic Core Volume Evaluated by Three CT Perfusion Software Packages in Acute Ischemic Stroke. J. Stroke Cereb. Dis. 2021, 30, 105872. [Google Scholar] [CrossRef]
  53. Ospel, J.M.; Cimflova, P.; Volny, O.; Qiu, W.; Hafeez, M.; Mayank, A.; Najm, M.; Chung, K.; Kashani, N.; Almekhlafi, M.A.; et al. Utility of Time-Variant Multiphase CTA Color Maps in Outcome Prediction for Acute Ischemic Stroke Due to Anterior Circulation Large Vessel Occlusion. Clin. Neuroradiol. 2021, 31, 783–790. [Google Scholar] [CrossRef] [PubMed]
  54. de la Rosa, E.; Sima, D.M.; Menze, B.; Kirschke, J.S.; Robben, D. AIFNet: Automatic Vascular Function Estimation for Perfusion Analysis Using Deep Learning. Med. Image Anal. 2021, 74, 102211. [Google Scholar] [CrossRef]
  55. de la Rosa, E.; Robben, D.; Sima, D.M.; Kirschke, J.S.; Menze, B. Differentiable Deconvolution for Improved Stroke Perfusion Analysis. In Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2020; Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 593–602. [Google Scholar]
  56. Pisani, L.; Mohammaden, M.; Bouslama, M.; Al-bayati, A.R.; Haussen, D.C.; Frankel, M.R.; Nogueira, R.G. Abstract P466: Comparison of Three Automated Ct Perfusion Software Packages for Thrombectomy Eligibility and Final Infarct Volume Prediction. Stroke 2021, 52, AP466. [Google Scholar] [CrossRef]
  57. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. Augmented Vascular Analysis 510(k) Premarket Notification Letter (K201369). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf20/K201369.pdf (accessed on 27 December 2021).
  58. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. Neuro.AI Algorithm 510(k) Premarket Notification Letter (K200750). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf20/K200750.pdf (accessed on 27 December 2021).
  59. Rava, R.A.; Snyder, K.V.; Mokin, M.; Waqas, M.; Zhang, X.; Podgorsak, A.R.; Allman, A.B.; Senko, J.; Shiraz Bhurwani, M.M.; Hoi, Y.; et al. Assessment of Computed Tomography Perfusion Software in Predicting Spatial Location and Volume of Infarct in Acute Ischemic Stroke Patients: A Comparison of Sphere, Vitrea, and RAPID. J. Neurointerv. Surg. 2021, 13, 130–135. [Google Scholar] [CrossRef]
  60. Ichikawa, S.; Yamamoto, H.; Morita, T. Comparison of a Bayesian Estimation Algorithm and Singular Value Decomposition Algorithms for 80-Detector Row CT Perfusion in Patients with Acute Ischemic Stroke. Radiol. Med. 2021, 126, 795–803. [Google Scholar] [CrossRef]
  61. Siegler, J.E.; Rosenberg, J.; Cristancho, D.; Olsen, A.; Pulst-Korenberg, J.; Raab, L.; Cucchiara, B.; Messé, S.R. Computed Tomography Perfusion in Stroke Mimics. Int. J. Stroke 2020, 15, 299–307. [Google Scholar] [CrossRef]
  62. Robert Ochs US Food and Drug Administration. BriefCase 510(k) Letter Premarket Notification Letter (K180647). 2018. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf18/K180647.pdf (accessed on 15 December 2021).
  63. Ojeda, P.; Zawaideh, M.; Mossa-Basha, M.; Haynor, D. The Utility of Deep Learning: Evaluation of a Convolutional Neural Network for Detection of Intracranial Bleeds on Non-Contrast Head Computed Tomography Studies. In Medical Imaging 2019: Image Processing; SPIE: Bellingham, WA, USA, 2019; Volume 10949, pp. 899–906. [Google Scholar]
  64. Ginat, D. Implementation of Machine Learning Software on the Radiology Worklist Decreases Scan View Delay for the Detection of Intracranial Hemorrhage on CT. Brain Sci. 2021, 11, 832. [Google Scholar] [CrossRef] [PubMed]
  65. Wismüller, A.; Stockmaster, L. A Prospective Randomized Clinical Trial for Measuring Radiology Study Reporting Time on Artificial Intelligence-Based Detection of Intracranial Hemorrhage in Emergent Care Head CT. In Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging; SPIE: Bellingham, WA, USA, 2020; Volume 11317, pp. 144–150. [Google Scholar]
  66. Ginat, D.T. Analysis of Head CT Scans Flagged by Deep Learning Software for Acute Intracranial Hemorrhage. Neuroradiology 2020, 62, 335–340. [Google Scholar] [CrossRef] [PubMed]
  67. Voter, A.F.; Meram, E.; Garrett, J.W.; Yu, J.-P.J. Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Intracranial Hemorrhage. J. Am. Coll. Radiol. 2021, 18, 1143–1152. [Google Scholar] [CrossRef]
  68. Kundisch, A.; Hönning, A.; Mutze, S.; Kreissl, L.; Spohn, F.; Lemcke, J.; Sitz, M.; Sparenberg, P.; Goelz, L. Deep Learning Algorithm in Detecting Intracranial Hemorrhages on Emergency Computed Tomographies. PLoS ONE 2021, 16, e0260560. [Google Scholar] [CrossRef]
  69. Rao, B.; Zohrabian, V.; Cedeno, P.; Saha, A.; Pahade, J.; Davis, M.A. Utility of Artificial Intelligence Tool as a Prospective Radiology Peer Reviewer—Detection of Unreported Intracranial Hemorrhage. Acad. Radiol. 2021, 28, 85–93. [Google Scholar] [CrossRef]
  70. Rava, R.A.; Seymour, S.E.; LaQue, M.E.; Peterson, B.A.; Snyder, K.V.; Mokin, M.; Waqas, M.; Hoi, Y.; Davies, J.M.; Levy, E.I.; et al. Assessment of an Artificial Intelligence Algorithm for Detection of Intracranial Hemorrhage. World Neurosurg. 2021, 150, e209–e217. [Google Scholar] [CrossRef] [PubMed]
  71. Ye, H.; Gao, F.; Yin, Y.; Guo, D.; Zhao, P.; Lu, Y.; Wang, X.; Bai, J.; Cao, K.; Song, Q.; et al. Precise Diagnosis of Intracranial Hemorrhage and Subtypes Using a Three-Dimensional Joint Convolutional and Recurrent Neural Network. Eur. Radiol. 2019, 29, 6191–6201. [Google Scholar] [CrossRef] [Green Version]
  72. Guo, D.; Wei, H.; Zhao, P.; Pan, Y.; Yang, H.-Y.; Wang, X.; Bai, J.; Cao, K.; Song, Q.; Xia, J.; et al. Simultaneous Classification and Segmentation of Intracranial Hemorrhage Using a Fully Convolutional Neural Network. In Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), Iowa City, IA, USA, 3–7 April 2020; pp. 118–121. [Google Scholar]
  73. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. Rapid ICH 510(k) Premarket Notification Letter (K193087). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf19/K193087.pdf (accessed on 27 December 2021).
  74. Heit, J.J.; Coelho, H.; Lima, F.O.; Granja, M.; Aghaebrahim, A.; Hanel, R.; Kwok, K.; Haerian, H.; Cereda, C.W.; Venkatasubramanian, C.; et al. Automated Cerebral Hemorrhage Detection Using RAPID. Am. J. Neuroradiol. 2021, 42, 273–278. [Google Scholar] [CrossRef]
  75. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. HealthICH 510(k) Premarket Notification Letter (K190424). 2019. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf19/K190424.pdf (accessed on 16 December 2021).
  76. Bar, A.; Havakuk, M.M.; Turner, Y.; Safadi, M.; Elnekave, E. Improved ICH Classification Using Task-Dependent Learning. In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy, 8–11 April 2019; pp. 1567–1571. [Google Scholar]
  77. Robert Ochs US Food and Drug Administration, Division of Radiological Health. Accipiolx 510(k) Premarket Notification Letter (K182177). 2018. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf18/K182177.pdf (accessed on 26 December 2021).
  78. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. DeepCT 510(k) Premarket Notification Letter (K182875). 2019. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf18/K182875.pdf (accessed on 16 December 2021).
  79. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. NinesAI 510(k) Premarket Notification Letter (K193351). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf19/K193351.pdf (accessed on 27 December 2021).
  80. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. QER 510(k) Premarket Notification Letter (K200921). 2020. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf20/K200921.pdf (accessed on 27 December 2021).
  81. Mills, T.T. US Food and Drug Administration, Division of Radiological Health. Viz ICH 510(k) Premarket Notification Letter (K210209). 2021. Available online: https://www.accessdata.fda.gov/cdrh_docs/pdf21/K210209.pdf (accessed on 27 December 2021).
  82. BrainQ. BrainQ Gets FDA Breakthrough Status for Its Device for Reducing Disability Following Stroke. Available online: https://www.prnewswire.com/news-releases/brainq-gets-fda-breakthrough-status-for-its-device-for-reducing-disability-following-stroke-301226735.html (accessed on 27 December 2021).
  83. BrainQ Gets FDA Breakthrough for Device to Reduce Disability Following. NS Medical Devices. 2021. Available online: https://www.nsmedicaldevices.com/news/brainq-ai-device/# (accessed on 27 December 2021).
  84. Wang, C.X.; Hilburn, I.A.; Wu, D.-A.; Mizuhara, Y.; Cousté, C.P.; Abrahams, J.N.H.; Bernstein, S.E.; Matani, A.; Shimojo, S.; Kirschvink, J.L. Transduction of the Geomagnetic Field as Evidenced from Alpha-Band Activity in the Human Brain. eNeuro 2019, 6, ENEURO.0483-18.2019. [Google Scholar] [CrossRef] [Green Version]
  85. Adaikkan, C.; Tsai, L.-H. Gamma Entrainment: Impact on Neurocircuits, Glia, and Therapeutic Opportunities. Trends Neurosci 2020, 43, 24–41. [Google Scholar] [CrossRef]
  86. Iaccarino, H.F.; Singer, A.C.; Martorell, A.J.; Rudenko, A.; Gao, F.; Gillingham, T.Z.; Mathys, H.; Seo, J.; Kritskiy, O.; Abdurrob, F.; et al. Gamma Frequency Entrainment Attenuates Amyloid Load and Modifies Microglia. Nature 2016, 540, 230–235. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  87. He, Q.; Pursiainen, S. An Extended Application ‘Brain Q’ Processing EEG and MEG Data of Finger Stimulation Extended from ‘Zeffiro’ Based on Machine Learning and Signal Processing. Cogn. Syst. Res. 2021, 69, 50–66. [Google Scholar] [CrossRef]
  88. Our Technology. Available online: https://brainqtech.com/our-technology (accessed on 15 December 2021).
  89. Weisinger, B.S.; Bornstein, N.M.; Shohami, E.; Segal, Y.; Alter, A.; Lifshitz, A.; Prasad, A.; Pandey, D. Abstract P194: Artificial Intelligence-Powered Non-Invasive and Frequency-Tuned Electromagnetic Field Therapy Improves Upper Extremity Motor Function in Sub-Acute Stroke Patients: A Pilot Randomized Controlled Trial. Stroke 2021, 52. [Google Scholar] [CrossRef]
  90. BrainQ Technologies Ltd. The Efficacy of a Frequency-Tuned Electromagnetic Field Treatment in Facilitating the Recovery of Subacute Ischemic Stroke Patients—A Pivotal Study; Clinicaltrials.gov: Bethesda, MD, USA, 2021.
  91. Efficacy of EMF BCI Based Device on Acute Stroke—Full Text View—ClinicalTrials.Gov. Available online: https://clinicaltrials.gov/ct2/show/NCT04039178 (accessed on 22 December 2021).
  92. Device. Available online: https://www.neurolutions.com/device (accessed on 16 December 2021).
  93. Bundy, D.T.; Souders, L.; Baranyai, K.; Leonard, L.; Schalk, G.; Coker, R.; Moran, D.W.; Huskey, T.; Leuthardt, E.C. Contralesional Brain–Computer Interface Control of a Powered Exoskeleton for Motor Recovery in Chronic Stroke Survivors. Stroke 2017, 48, 1908–1915. [Google Scholar] [CrossRef] [PubMed]
  94. RapidAI Achieves Record Momentum. Available online: https://www.rapidai.com/press-release/rapidai-achieves-record-momentum (accessed on 8 May 2022).
  95. Feldman, A. Next Billion-Dollar Startups: How Viz.Ai Helps Hospitals Treat Stroke Patients Earlier. Available online: https://www.forbes.com/sites/amyfeldman/2022/01/03/next-billion-dollar-startups-how-vizai-helps-hospitals-treat-stroke-patients-earlier-video/ (accessed on 8 May 2022).
  96. Dickson, R.; Nedelcut, A.; Seupaul, R.; Hamzeh, M. STOP STEMI©-A Novel Medical Application to Improve the Coordination of STEMI Care: A Brief Report On Door-to-Balloon Times After Initiating the Application. Crit. Pathw. Cardiol. 2014, 13, 85–88. [Google Scholar] [CrossRef] [Green Version]
  97. Ali, S.; Smith, K.A. On Learning Algorithm Selection for Classification. Appl. Soft Comput. 2006, 6, 119–138. [Google Scholar] [CrossRef]
  98. Chicco, D. Ten Quick Tips for Machine Learning in Computational Biology. BioData Min. 2017, 10, 35. [Google Scholar] [CrossRef] [Green Version]
  99. Kotthoff, L.; Gent, I.P.; Miguel, I. An Evaluation of Machine Learning in Algorithm Selection for Search Problems. AI Commun. 2012, 25, 257–270. [Google Scholar] [CrossRef]
  100. Raschka, S. Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning. arXiv 2020, arXiv:1811.12808[cs, stat]. [Google Scholar]
  101. Kuo, W.; Häne, C.; Mukherjee, P.; Malik, J.; Yuh, E.L. Expert-Level Detection of Acute Intracranial Hemorrhage on Head Computed Tomography Using Deep Learning. Proc. Natl. Acad. Sci. USA 2019, 116, 22737–22745. [Google Scholar] [CrossRef] [Green Version]
  102. Zihni, E.; Madai, V.I.; Livne, M.; Galinovic, I.; Khalil, A.A.; Fiebach, J.B.; Frey, D. Opening the Black Box of Artificial Intelligence for Clinical Decision Support: A Study Predicting Stroke Outcome. PLoS ONE 2020, 15, e0231166. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  103. Yang, G.; Ye, Q.; Xia, J. Unbox the Black-Box for the Medical Explainable AI via Multi-Modal and Multi-Centre Data Fusion: A Mini-Review, Two Showcases and Beyond. Inf. Fusion 2022, 77, 29–52. [Google Scholar] [CrossRef]
  104. Ghassemi, M.; Oakden-Rayner, L.; Beam, A.L. The False Hope of Current Approaches to Explainable Artificial Intelligence in Health Care. Lancet Digit. Health 2021, 3, e745–e750. [Google Scholar] [CrossRef] [PubMed]
  105. Zhang, Y.; Weng, Y.; Lund, J. Applications of Explainable Artificial Intelligence in Diagnosis and Surgery. Diagnostics 2022, 12, 237. [Google Scholar] [CrossRef] [PubMed]
  106. Hassan, A.E. New Technology Add-On Payment (NTAP) for Viz LVO: A Win for Stroke Care. J. Neurointerv. Surg. 2021, 13, 406–408. [Google Scholar] [CrossRef]
  107. RapidAI Among the First Stroke Imaging Companies with Software Approved for Medicare New Technology Add-on Payment. Available online: https://www.businesswire.com/news/home/20201001005423/en/RapidAI-Among-the-First-Stroke-Imaging-Companies-with-Software-Approved-for-Medicare-New-Technology-Add-on-Payment (accessed on 25 December 2021).
Jcm 12 03755 g001 550
Figure 1. Literature and web search methodologies. A Google search was conducted using a compound search with terms related to stroke, artificial intelligence/machine learning, and FDA approval. Seventy links were evaluated based on stopping criteria, which resulted in the discovery of 7 unique technologies. A total of 343 technologies were evaluated from an FDA database, which resulted in the discovery of 21 unique technologies. The final list of 22 technologies was created after excluding six overlaps from the two searches. CVA: Cerebrovascular Accident.
Figure 1. Literature and web search methodologies. A Google search was conducted using a compound search with terms related to stroke, artificial intelligence/machine learning, and FDA approval. Seventy links were evaluated based on stopping criteria, which resulted in the discovery of 7 unique technologies. A total of 343 technologies were evaluated from an FDA database, which resulted in the discovery of 21 unique technologies. The final list of 22 technologies was created after excluding six overlaps from the two searches. CVA: Cerebrovascular Accident.
Jcm 12 03755 g001
Jcm 12 03755 g002 550
Figure 2. Reported technology performance from the literature. Various performance metrics (x-axis; using human performance as the gold standard) are reported for (A) LVO and ASPECT technologies, (B) CTP technologies, and (C) ICH technologies. Bars represent the minimum and maximum reported values for each metric. Data are depicted as a single point when only one value is reported in the literature. Not every technology has data for all performance metrics. Variation in performance was generally higher for LVO/ASPECT technologies compared to ICH technologies. More performance quantification is needed for CTP technologies. ASPECT: Alberta Stroke Program Early CT Score; AUC: Area Under the Receiver Operating Curve; CTP: Computed Tomography Perfusion; ICH: Intracranial Hemorrhage; LVO: Large Vessel Occlusion; NPV: Negative Predictive Value; PPV: Positive Predictive Value. Metrics for Rapid ASPECTS are related to calculating ASPECT scores, while other technologies in (A) are evaluated on LVO detection.
Figure 2. Reported technology performance from the literature. Various performance metrics (x-axis; using human performance as the gold standard) are reported for (A) LVO and ASPECT technologies, (B) CTP technologies, and (C) ICH technologies. Bars represent the minimum and maximum reported values for each metric. Data are depicted as a single point when only one value is reported in the literature. Not every technology has data for all performance metrics. Variation in performance was generally higher for LVO/ASPECT technologies compared to ICH technologies. More performance quantification is needed for CTP technologies. ASPECT: Alberta Stroke Program Early CT Score; AUC: Area Under the Receiver Operating Curve; CTP: Computed Tomography Perfusion; ICH: Intracranial Hemorrhage; LVO: Large Vessel Occlusion; NPV: Negative Predictive Value; PPV: Positive Predictive Value. Metrics for Rapid ASPECTS are related to calculating ASPECT scores, while other technologies in (A) are evaluated on LVO detection.
Jcm 12 03755 g002
Table
Table 1. List of 22 unique AI/ML-enabled, FDA-approved technologies indicated for diagnosis and/or management of stroke. Twenty technologies are indicated for stroke diagnosis and two for post-stroke rehabilitation, with approval dates ranging from February 2018 to April 2021. Eighteen companies developed these technologies, with 11/18 (61%) headquartered outside the United States. Generally, the technologies indicated for stroke diagnosis utilize convolutional neural networks as their underlying algorithm. CTA: Computed Tomography Angiography; CTP: Computed Tomography Perfusion; ICH: Intracranial Hemorrhage; LVO: Large Vessel Occlusion. * RAPID is indicated for both LVO and CTP analysis. ** CINA is indicated for both LVO and ICH analyses. 510(k) refers to a premarket submission made to the FDA demonstrating the proposed device to be as safe and effective as a legally marketed device. 513(f)(2) (De Novo) classification describes devices that are considered for Class I (low-to-moderate risk) or II (moderate-to-high risk) categorization, either after receiving a “not substantially equivalent” determination post-510(k) submission or in cases where there is no legally marketed device to determine substantial equivalence. A breakthrough status designation aims to expedite the development, assessment, and review of a medical device to provide patients with timely access to such a device while maintaining FDA statutory standards in the approval process.
Table 1. List of 22 unique AI/ML-enabled, FDA-approved technologies indicated for diagnosis and/or management of stroke. Twenty technologies are indicated for stroke diagnosis and two for post-stroke rehabilitation, with approval dates ranging from February 2018 to April 2021. Eighteen companies developed these technologies, with 11/18 (61%) headquartered outside the United States. Generally, the technologies indicated for stroke diagnosis utilize convolutional neural networks as their underlying algorithm. CTA: Computed Tomography Angiography; CTP: Computed Tomography Perfusion; ICH: Intracranial Hemorrhage; LVO: Large Vessel Occlusion. * RAPID is indicated for both LVO and CTP analysis. ** CINA is indicated for both LVO and ICH analyses. 510(k) refers to a premarket submission made to the FDA demonstrating the proposed device to be as safe and effective as a legally marketed device. 513(f)(2) (De Novo) classification describes devices that are considered for Class I (low-to-moderate risk) or II (moderate-to-high risk) categorization, either after receiving a “not substantially equivalent” determination post-510(k) submission or in cases where there is no legally marketed device to determine substantial equivalence. A breakthrough status designation aims to expedite the development, assessment, and review of a medical device to provide patients with timely access to such a device while maintaining FDA statutory standards in the approval process.
DeviceCompany NameHeadquartersFDA Approval NumberType of ApprovalIndicationDate of Approval
ContaCT (Viz LVO)Viz.aiSan Francisco, CA, USADEN170073513(f)(2)(De Novo)Analyze acute CTA to identify LVO February 2018
Viz CTPViz.aiSan Francisco, CA, USAK180161510(k)Analyze brain tissue perfusion parameters on CTPApril 2018
BriefCaseAidoc MedicalIsraelK180647510(k)ICH detection in non-contrast CTAugust 2018
AccipiolxMaxQ AlIsraelK182177510(k)ICH detection in non-contrast CTOctober 2018
Vitrea CT Brain PerfusionVital ImagesMinnetonka, MN, USAK181247510(k)Visualize apparent blood perfusion in brain tissue affected by acute stroke on CTNovember 2018
RAPID *iSchemaViewGolden, CO, USAK182130510(k)Identification of CTP, CTA, and MRI images consistent with strokeDecember 2018
HealthICHZebra Medical Vision (now Nanox AI)IsraelK190424510(k)Aid clinical assessment of non-contrast head CT with features suggestive of ICHJune 2019
DeepCTDeep01 LimitedTaiwanK182875510(k)ICH detection in non-contrast CTJuly 2019
Icobrain-CTPIcometrixBelgiumK192962510(k)Image analysis of brain CT perfusion scansFebruary 2020
Rapid ICHiSchemaViewGolden, CO, USAK193087510(k)ICH detection in non-contrast CTMarch 2020
CuraRad-ICHCuraCloudSeattle, WA, USAK192167510(k)ICH detection in non-contrast CTApril 2020
NinesAINinesPalo Alto, VA, USAK193351510(k)ICH detection in non-contrast CTApril 2020
CINA **AVICENNA.AIFranceK200855510(k)ICH in head CT and LVO in head CT angiographyJune 2020
Rapid ASPECTSiSchemaViewGolden, CO, USAK200760510(k)ASPECT scoring in patients with known MCA or ICA occlusionsJune 2020
QERQure.Ai TechnologiesIndiaK200921510(k)ICH detection in non-contrast CTJune 2020
Augmented Vascular AnalysisSee-Mode TechnologiesSingaporeK201369510(k)Predict stroke risk from vascular ultrasoundSeptember 2020
HALONICo-Lab B.V.AmsterdamK200873510(k)LVO identification in anterior circulation (ICA, M1 or M2) from CT angiogramNovember 2020
FastStroke, CT Perfusion 4DGE Medical Systems SCSFranceK193289510(k)CT Perfusion 4D: Perfusion abnormalities from contrast CTFastStroke: Stroke detection from CT (e.g., non-constast, angiogram)November 2020
Neuro.Al AlgorithmTeraReconDurham, NC, USAK200750510(k)Detect changes in brain perfusion from CT or MRINovember 2020
BrainQBrainQIsrael-Breakthrough statusReduce disability post-strokeFebruary 2021
Viz ICHViz.aiSan Francisco, CA, USAK210209510(k)Analyze acute non-contrast CT of brain, notify specialist of suspected ICHMarch 2021
IpsiHand Upper Extremity Rehabilitation SystemNeurolutionsSanta Cruz, CA, USA-Breakthrough statusPost-stroke rehabilitationApril 2021
Table
Table 2. Five technologies are indicated to diagnose LVO and calculate the Alberta Stroke Program Early CT Score (ASPECTS). The majority of studies assessing these technologies were retrospective. Algorithm performance either met or exceeded human performance, and implementation of the solutions has improved clinical workflows and patient outcomes. Difficulties with LVO identification were occasionally seen with vessel anatomical variation. Human performance generally continues to be the gold standard for evaluating these algorithms. ICA: Internal Carotid Artery; MCA: Middle Cerebral Artery. * Indicates metric was extrapolated from available data.
Table 2. Five technologies are indicated to diagnose LVO and calculate the Alberta Stroke Program Early CT Score (ASPECTS). The majority of studies assessing these technologies were retrospective. Algorithm performance either met or exceeded human performance, and implementation of the solutions has improved clinical workflows and patient outcomes. Difficulties with LVO identification were occasionally seen with vessel anatomical variation. Human performance generally continues to be the gold standard for evaluating these algorithms. ICA: Internal Carotid Artery; MCA: Middle Cerebral Artery. * Indicates metric was extrapolated from available data.
DeviceAuthor, YearLevel of EvidenceDataset CharacteristicsSample Size (Scans)AUCPPVNPVAccuracySensitivitySpecificityOther Metrics/Comments
Viz LVOHassan et al., 2020 [22]ProspectiveProprietary,
Single Center		43------Viz LVO reduced median CTA time at primary center to door-in at comprehensive center by an average of 22.5 min. Neuro-ICU stays were reduced by 2.5 days.
Yahav-Dovrat et al., 2021 [23]ProspectiveProprietary,
Single Center		1167-65%99%94%81%96%-
Morey et al., 2021 [21]RetrospectiveProprietary,
Single Center		55------Viz LVO reduced median door-to-neurovascular team notification time from 40 to 25 min.
Rodrigues et al., 2021 [24]RetrospectiveProprietary,
Single Center		6100.8893%79%88%88%89%Algorithm had similar performance across ICA-T, MCA-M1, and MCA-M2 occlusions. Mean run time was ~3 min.
RAPID (LVO)Adhya et al., 2021 [25]RetrospectiveProprietary,
Multicenter		310-23–75%--80%-CT to groin puncture time was lower after implementation of RAPID (93 min vs. 68 min).
Amukotuwa et al., 2019 [26]RetrospectiveProprietary,
Single Center		4770.77–0.8614–58%97–99%-86–94%68–81%Median scan analysis time was roughly 160 s.
CINAMcLouth et al., 2021 [27]RetrospectiveProprietary,
Multicenter		378-86–98%98–99%98%98%98%In the detection of LVO subtypes (i.e., at distal internal carotid artery, middle cerebral artery M1 segment, proximal middle cerebral artery M2 segment, distal middle cerebral artery M2 segment), the CINA algorithm demonstrated an accuracy of 97%, sensitivity of 94.3%, and specificity of 97.4%.
Rava et al., 2021 [28]RetrospectiveProprietary,
Single Center		303-99%64%81%73%98%Scan processing time was ~70 s. The algorithm identified ICA, M1 MCA, and M2 MCA occlusions.
HALOLuijten et al., 2021 [29]ProspectiveMR CLEAN registry & PRESTO study,
Multicenter		17560.7547%91%76% *72-89%78%Performance varied considerably based on location of occlusion.
Rapid ASPECTSLasocha et al., 2020 [30]RetrospectiveProprietary,
Single Center		100------Exact ASPECT score agreement between RAPID and manual methods was poor, but crossing of threshold for reperfusion therapy was characterized by an 80% match.
Hoelter et al., 2020 [31]RetrospectiveProprietary,
Single Center		1310.73-----Correlation between ASPECT scores of experts and RAPID was high (r = 0.78)
Maegerlein et al., 2019 [32]RetrospectiveProprietary,
Single Center		100------In acute stroke of the middle cerebral artery, RAPID-calculated ASPECT score had better agreement with predefined consensus scores than neuroradiologists overall (κ = 0.9 vs. κ = 0.57, respectively), and particularly in the time interval of 1 to 4 h between symptom onset and imaging.
Al-Kawaz et al., 2021 [33] RetrospectiveProprietary,
Single Center		64------Use of the RAPID mobile app (which includes Rapid ASPECTS functionality) decreased door to groin puncture times by 33 min compared to patients treated pre-app and improved scores on National Institutes of Health Stroke Scale 24 h after procedure (12.1 vs. 8.0) and at discharge (11.8 vs. 7.8)
Albers et al., 2019 [34]RetrospectiveGAMES-RP trial,
Multicenter		65---73%--RAPID ASPECTS was more accurate than clinicians (73% vs. 56%) in identifying early ischemia on DWI.
Mansour et al., 2020 [35]RetrospectiveProprietary,
Single Center		122------Automated ASPECT score by the algorithm performed equally to scoring by neuroradiologists (κ = 0.8).
Table
Table 3. Seven technologies are indicated to analyze CTP images. The majority of studies were retrospective. Algorithms are able to predict final infarct volume and/or assess the quality of collateral perfusion, and algorithm performance met or exceeded human performance in binary and multiclass classification. ICH location (e.g., under the calvaria) and anatomical variations (e.g., calcification of the falx) reduced algorithm performance. Human performance generally continues to be the gold standard for evaluating these algorithms. CBF: Cerebral Blood Flow; CBV: Cerebral Blood Volume; MTT: Mean Transit Time; SVD: Singular Value Decomposition.
Table 3. Seven technologies are indicated to analyze CTP images. The majority of studies were retrospective. Algorithms are able to predict final infarct volume and/or assess the quality of collateral perfusion, and algorithm performance met or exceeded human performance in binary and multiclass classification. ICH location (e.g., under the calvaria) and anatomical variations (e.g., calcification of the falx) reduced algorithm performance. Human performance generally continues to be the gold standard for evaluating these algorithms. CBF: Cerebral Blood Flow; CBV: Cerebral Blood Volume; MTT: Mean Transit Time; SVD: Singular Value Decomposition.
DeviceAuthor, YearLevel of EvidenceDataset CharacteristicsSample Size (Scans)AUCPPVNPVAccuracySensitivitySpecificityOther Metrics/Comments
Vitrea CT Brain PerfusionRava et al., 2020 [47]RetrospectiveProprietary,
Single Center		105------In estimating infarct volume, Spearman correlation coefficient between Vitrea and DWI/FLAIR ranged from 0.71 to 0.77. Vitrea outperformed RAPID.
Rava et al., 2020 [48]RetrospectiveProprietary,
Single Center		107------In estimating infarct volume, Spearman correlation coefficient between different algorithms within Vitrea (i.e., Bayesian and Singular Value Decomposition) and FLAIR MRI was 0.98 vs. 0.76-0.87 between RAPID and FLAIR MRI.
Rava et al., 2021 [49]RetrospectiveProprietary,
Single Center		63-63–72%-----
Rava et al., 2021 [59]RetrospectiveProprietary,
Single Center		108---96–98%60–62%98–99%Vitrea overestimated infarct volume, but provided the most accurate penumbra assessment for patients treated conservatively.
Ichikawa et al., 2021 [60]Retrospective
Proprietary,
Single Center		36------Vitrea’s Bayesian algorithm had better delineation of abnormal perfusion areas and estimation of infarct volume compared to the SVD implementation.
RAPID (CTP)Hokkinen et al., 2021 [40]RetrospectiveProprietary,
Single Center		89------In patients presenting 6 to 24 hours from onset of symptoms, CTP-RAPID’s estimate of infarct volume correlated with follow-up imaging (r = 0.82). Correlation decreased (r = 0.58) in patients presenting 0 to 6 hours after symptom onset.
Wouters et al., 2021 [42]Randomized Controlled TrialMR CLEAN trial & CRISP study,
Multicenter		127------A new deep learning CNN model outperformed RAPID in predicting final infarct volume.
Potreck et al., 2021 [43]RetrospectiveSimulation53------Head motion during CT perfusion acquisition can impact infarct core estimates.
Bouslama et al., 2021 [44]RetrospectiveProprietary,
Single Center		479------RAPID had moderate correlation with final infarct volumes (r = 0.42–0.44).
Siegler et al., 2020 [61]RetrospectiveMulti-site registry,
Multicenter		4100.69---62%72%Stroke mimics can show abnormalities on RAPID CT analysis.
Kim et al., 2019 [45]ProspectiveProprietary,
Single Center		296---89–100%--Interclass correlation between RAPID and manual measurements of infarct volume were 0.98, with RAPID underestimating volumes by ~2 mL on average.
FastStroke/CT Perfusion 4DVerdolotti et al., 2020 [51]RetrospectiveProprietary,
Single Center		86------Algorithm is comparable in efficacy to the status quo in evaluating collateral circulation, but has simpler workflows and faster turnaround times, making use easier for radiologists.
Ospel et al., 2021 [53]ProspectivePRove-IT cohort study,
Multicenter		2850.63–0.76-----Time-variant multiphase CTA (mCTA) maps produced by the software improved prediction of good outcomes and performed comparably to conventional mCTA in predicting infarct volume.
Liu et al., 2021 [52]RetrospectiveProprietary,
Single Center		82------CT Perfusion 4D had ICC of 0.95 compared to RAPID in predicting core volumes. The algorithm also performed well for volumes ≤ 70 mL
Icobrain-CTPde la Rosa et al., 2021 [54]RetrospectivePublic ISLES18 stroke database156------Icobrain uses a CNN that does not need user input in the form of thresholding to assess perfusion. Estimations of penumbra volume using CBF, CBV, and MTT had strong correlation with assessments by radiologists.
de la Rosa et al., 2021 [55]RetrospectivePublic ISLES18 stroke database1560.86-----Icobrain performed comparably to expert estimates of cerebral blood flow based on 4D CTP scans.
Viz CTPPisani et al., 2021 [56]ProspectiveProprietary database otherwise unspecified242------Viz CTP performed well in predicting final infarct volume (r = 0.601).
Augmented Vascular Analysis
Neuro.Al Algorithm
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Chandrabhatla, A.S.; Kuo, E.A.; Sokolowski, J.D.; Kellogg, R.T.; Park, M.; Mastorakos, P. Artificial Intelligence and Machine Learning in the Diagnosis and Management of Stroke: A Narrative Review of United States Food and Drug Administration-Approved Technologies. J. Clin. Med. 2023, 12, 3755. https://doi.org/10.3390/jcm12113755

AMA Style

Chandrabhatla AS, Kuo EA, Sokolowski JD, Kellogg RT, Park M, Mastorakos P. Artificial Intelligence and Machine Learning in the Diagnosis and Management of Stroke: A Narrative Review of United States Food and Drug Administration-Approved Technologies. Journal of Clinical Medicine. 2023; 12(11):3755. https://doi.org/10.3390/jcm12113755

Chicago/Turabian Style

Chandrabhatla, Anirudha S., Elyse A. Kuo, Jennifer D. Sokolowski, Ryan T. Kellogg, Min Park, and Panagiotis Mastorakos. 2023. "Artificial Intelligence and Machine Learning in the Diagnosis and Management of Stroke: A Narrative Review of United States Food and Drug Administration-Approved Technologies" Journal of Clinical Medicine 12, no. 11: 3755. https://doi.org/10.3390/jcm12113755

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop