An Overview of Open Source Deep Learning-Based Libraries for Neuroscience

Abstract

In recent years, deep learning has revolutionized machine learning and its applications, producing results comparable to human experts in several domains, including neuroscience. Each year, hundreds of scientific publications present applications of deep neural networks for biomedical data analysis. Due to the fast growth of the domain, it could be a complicated and extremely time-consuming task for worldwide researchers to have a clear perspective of the most recent and advanced software libraries. This work contributes to clarifying the current situation in the domain, outlining the most useful libraries that implement and facilitate deep learning applications for neuroscience, allowing scientists to identify the most suitable options for their research or clinical projects. This paper summarizes the main developments in deep learning and their relevance to neuroscience; it then reviews neuroinformatic toolboxes and libraries collected from the literature and from specific hubs of software projects oriented to neuroscience research. The selected tools are presented in tables detailing key features grouped by the domain of application (e.g., data type, neuroscience area, task), model engineering (e.g., programming language, model customization), and technological aspect (e.g., interface, code source). The results show that, among a high number of available software tools, several libraries stand out in terms of functionalities for neuroscience applications. The aggregation and discussion of this information can help the neuroscience community to develop their research projects more efficiently and quickly, both by means of readily available tools and by knowing which modules may be improved, connected, or added.

1. Introduction

In the last decade, deep learning has taken over most classic approaches in machine learning, computer vision, and Natural Language Processing (NLP) research, showing unprecedented versatility and matching or surpassing the performances of human experts in narrow tasks. The recent growth of deep learning applications in several domains, including neuroscience, consequently offers numerous open-source software opportunities for researchers. Mapping available resources can allow for faster and more precise exploitation. Neuroscience is a diversified field on its own, as much for the objects and scales it focuses on as for the types of data it relies on. The discipline is also historically tied to developments in electrical, electronic, and information technology. Modern neuroscience relies on computerization in many aspects of data generation, acquisition, and analysis. Statistical and machine learning techniques already empower many software packages that have become de facto standards in several subfields of neuroscience, such as Principal and Independent Component Analysis (PCA, ICA) in electroencephalography and neuroimaging, to name a few. Concurrently, the rich and rapidly evolving taxonomy of Deep Neural Networks (DNNs) is becoming both an opportunity and a hindrance. On the one hand, currently, open-source deep learning libraries allow an increasing number of applications and studies in neuroscience. On the other hand, the adoption of available methods is slowed down by a lack of standards, reference frameworks, and established workflows. Scientific communities whose primary focus or background is not in machine learning engineering may be left partially aside from the ongoing Artificial Intelligence (AI) gold rush. For such reasons, it is fundamental to provide an overview of open-source libraries and toolkits. Framing a panorama could help researchers in selecting ready-made tools and solutions when convenient and aid them in pointing out problems and filling in the blanks with new applications. This work would contribute to advancing the community’s possibilities, reducing the workload for researchers to exploit deep learning, and allowing neuroscience to benefit from its most recent advancements.

The rest of the paper is organized as follows: first in the Section 2, a historical perspective of the rise of deep learning in the last decade, then, a general presentation of the vast field of neuroscience followed by a definition of neuroinformatics and the role open source culture and deep learning would serve in it; subsequently, in the Section 3, the methodology to collect and present the libraries collection, and the most prominent features are discussed; the Section 4 shows the tables with libraries information; lastly, discussions and final remarks are offered to the readers (Section 5 and Section 6).

2. Background

2.1. Deep Learning

Deep learning has contributed many of the best solutions to problems in its parent field, machine learning, thanks to its theoretical and technological achievements that unlocked its intrinsic versatility. Machine learning is the study of computer algorithms that tackle problems without complete access to predefined rules or analytical, closed-form solutions. The algorithms often require a training phase to adjust parameters and satisfy internal or external constraints (e.g., of exactness, approximation, or generality) on dedicated data for which solutions might be already known. Machine learning comprises a wide array of statistical and mathematical methods, including Artificial Neural Networks (ANNs), biologically inspired systems that connect inputs and outputs through simple computing units (neurons), which act as function approximators. Each unit implements a nonlinear function of the weighted sum of its inputs; thus, the output of the whole ANN is a composite function, as formally intended in mathematics. The networks of neurons are most often layered and “feed-forward”, meaning that units from any layer only output results to units in subsequent layers. The width of a layer refers to its neuron count, while the depth of a network refers to its layer count. The typical architecture instantiating the above characteristics is the MultiLayer Perceptron [1] (MLP). Universal approximation theorems [2,3] ensure that whenever a nonlinear network, such as the MLP, is either bound in width and unbound in depth or vice versa, its weights can then be set to represent virtually any function (i.e., a wide variety of function families). The training problem thus consists of building networks with sets of weights so to instantiate or approximate the function that would solve the assigned task or that represents the input-output relation. This search is not trivial: it can be framed as the optimization problem for a function over the ANN weights. Such functions, typically called “loss function”, associates the “errors” made on the training data to the neural net parameters (its weights), acting as a total performance score. Approaching local minima of the loss function and improving the network performance on the training data is the prerequisite to generalizing on real-world and unseen data. Deep learning is concerned with the use of deep ANNs, namely characterized by depth, stacking several intermediate (hidden) layers between input and output units. As mentioned above, with other dimensions being equal, the depth increases the representational power of ANNs and, more specifically, aims at modeling complicated functions as meaningful compositions of simpler ones. As with their biological counterparts [4], depth is supposed to manage hierarchies of features from larger input portions, capturing characteristics often inherent to real-world objects and effective in modeling actual data. Overall, depth is one of the key features that allowed us to overcome historical limits [5] of simpler ANNs such as the Perceptron. At the same time, depth comes with numerical and methodological hardships in model training. Part of the difficulties arises as the search space for the optimal set of parameters grows considerably with the number of layers (and their width as well). Other issues are strictly numerical since the training algorithms include long computation chains that may affect the stability of training and learning. Hence, new or rediscovered ideas in training protocols and mathematical optimization (e.g., applying the “backpropagation of errors” algorithm to neural nets [6]) played an important role through times when the scientific interest and hopes in ANNs faded (so-called “AI winters”), paving the way for later advancement. The main drivers for the latest success of deep neural networks are of varied nature and can be schematized as technical and human-related factors. On a technical side, deep learning has profited from [7]:

• The datafication of the world, i.e., the growing availability of (Big) data
• The diffusion of Graphical Processing Units (GPUs) as hardware tools.

To outperform classic machine learning models, deep neural networks often require larger quantities of data samples. Such data hunger and high parameter count contribute to the high requirements of deep models in terms of memory, number of operations, and computation time. Training models with highly parallelized and smartly scheduled computations gained momentum thanks to GPUs. In 2012 a milestone exemplified both the above technical aspects when AlexNet [8], a deep Convolutional Neural Network (CNN) based on ideas from Fukushima [4] and LeCun [9,10], won the ImageNet Large Scale Visual Recognition Challenge after being trained using two GPUs [11]. Since then, deep learning has brought new outstanding results in various tasks and domains, processing different data types. Nowadays, deep networks can work on images, video, audio, text, and speech data, time series and sequences, graphs, and more; the main tasks consist of classification, prediction, or estimating the probability density of data distributions, with the possibility of modifying, completing the input, or even generating new instances. On a more sociological side, the drivers of deep learning success can be related to the synergy of big tech companies, advanced research centers, and developer communities [12]. Investments of economic and scientific resources in relatively independent, collective projects, such as open-source libraries, frameworks, and APIs (Application Programming Interfaces), have offered varied tools adapted to multiple specific situations and objectives, exploiting horizontal organization [13] and mixing top-down and bottom-up approaches. It is difficult to imagine a rapid rise of successful endeavors without both active communities and the technical means to incorporate and manage lower-level aspects. In fact, applying deep learning to a relevant problem in any research field requires, in addition to specific domain knowledge, a vast background of statistical, mathematical, and programming notions and skills. The tools that support scientists and engineers in focusing on their main tasks encompass the languages to express numerical operations on GPUs, such as CUDA [14] and cuDNN [15] by NVIDIA, as well as the frameworks to design models, like TensorFlow [16] and Keras [17] by Google, and PyTorch by Meta [18], or the supporting strategies to build data pipelines. In particular, PyTorch, TensorFlow, and Keras offer the building blocks for model design. These frameworks comprise the mathematical operations and functions that deep learning models perform during training and at test time. The functions can be treated as modular objects, stacked one upon another, or connected in more complex ways. The data are input and processed through these objects-functions chains to return corresponding outputs. On a higher level, one can ignore computational and mathematical details as long as the role and effect of such components are understood. On a lower level, these frameworks allow the experts in the community to introduce and share novel custom objects and operations that push forward deep learning research. Data loading and preprocessing modules are included, as well as many pre-trained deep learning models, enhancing the framework’s adaptability and usability. Many deep learning achievements are relevant to biomedical and clinical research, and the above-presented tools have enabled explorations of the capabilities of deep neural networks with neuroscience and biomedical data. Fuller exploitation and routine employment of modern algorithms are yet to come, both in research and clinical practice. This process would accelerate by popularizing, democratizing, and jointly developing models, improving their usability, and expanding their environments, i.e., by wrapping solutions into libraries and shared frameworks.

2.2. Neuroscience

As per the journal Nature, <<neuroscience is a multidisciplinary science that is concerned with the study of the structure and function of the nervous system. It encompasses the evolution, development, cellular and molecular biology, physiology, anatomy, and pharmacology of the nervous system, as well as computational, behavioral, and cognitive neuroscience>> [19]. In summary, neuroscience investigates:

• The evolutionary and individual development of the nervous system;
• The cellular and molecular biology that characterizes neurons and glial cells;
• The physiology of living organisms and the role of the nervous system in the homeostatic function;
• The anatomy, i.e., the identification and description of the system’s structures;
• Pharmacology, i.e., the effect of chemicals of external origin on the nervous system, their interactions with endogenous molecules;
• The computational features of the brain and nerves, how information is processed, which mathematical and physical models best predict and approximate the behavior of neurons;
• Cognition, the mental processes at the intersection of psychology and computational neuroscience;
• Behavior as a phenomenon rooted in genetics, development, mental states, and so forth.

Overall, given the wide range of phenomena and the apparatus it investigates, neuroscience research is profoundly multi-modal. Data range from sequences or signals (e.g., electromyography (EMG), electroencephalography (EEG), eye-tracking, genetic sequencing), to 2D/3D images (e.g., Magnetic Resonance Imaging (MRI), X-rays, tomography, histopathology microscopy, eye fundus photography) or videos. Tabular data and text data are also common in this field, from clinical reports and anamneses to surveys, test scores, and inspections of cognitive and sensorimotor functions (e.g., the National Institute of Health (NIH) Stroke Scale test scores [20]), and more.

2.3. Neuroinformatics

Neuroscience is evolving into a data-centric discipline. Modern research heavily depends on human researchers as well as machine agents to store, manage, and process computerized data from the experimental apparatus to the end stage. Before delving into the specifics of artificial neural networks applied to the study of biological neural systems, it is useful to outline the broader concepts of neuroinformatics, regarding data and coding, especially in the light of open culture. According to the International Neuroinformatics Coordinating Facility (INCF), <<neuroinformatics is a research field devoted to the development of neuroscience data and knowledge bases together with computational models and analytical tools for sharing, integration, and analysis of experimental data and advancement of theories about the nervous system function.>> [21]. Given the relevance of neuroinformatics to neuroscience, supporting open and reproducible science implies and requires attention to standards and best practices regarding open data and code. The INCF itself is an independent organization devoted to validating and promoting such standards and practices, interacting with the research communities [22] and aiming at the “FAIR principles for scientific data management and stewardship” [23]. FAIR principles consist in:

• Being Findable, registered and indexed, searchable, richly described in metadata;
• Being Accessible, through open, free, universally implementable protocols;
• Being Interoperable, with appropriate standards for metadata in the context of knowledge representation;
• Being Reusable, clearly licensed, well described, relevant to a domain, and meeting community standards.

Among free and open resources, several software and organized packages integrating pre-processing and data analysis workflows for neuroimaging and signal processing became the reference for worldwide researchers in neuroscience.

Such tools allow us to perform scientific research in neuroscience easily in solid and repeatable ways. It can be useful to mention, for neuroimaging, Freesurfer (https://surfer.nmr.mgh.harvard.edu/) [24] and FSL (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki) [25] that are standalone softwares, and the MATLAB-connected SPM (https://www.fil.ion.ucl.ac.uk/spm/) [26]. In the domain of signal processing, examples are EEGLAB (https://sccn.ucsd.edu/eeglab/index.php) [27], Brainstorm (https://neuroimage.usc.edu/brainstorm/Introduction) [28], PaWFE (http://ninapro.hevs.ch/node/229) [29], all MATLAB related yet free and open, and MNE (https://mne.tools/stable/index.html) [30], that runs on Python. Regarding applications for neurorobotics and Brain Computer Interfaces (BCIs), a recent open source platform can be found in ROS-neuro (https://github.com/rosneuro) [31]. All URLs accessed at date 22nd of November 2022. Interested readers can find lists of open resources for computational neuroscience (including code, data, models, repositories, textbooks, analysis, simulation, and management software) at Open Computational Neuroscience Resource (https://github.com/asoplata/open-computational-neuroscience-resources) (by Austin Soplata), and at Open Neuroscience (https://open-neuroscience.com/). Additional software resources oriented to neuroinformatics in general, but not necessarily open, can also be found as indexed at “COMPUTATIONAL NEUROSCIENCE on the Web” (https://compneuroweb.com/sftwr.html) (by Jim Perlewitz).

2.4. Bringing Deep Learning to the Neurosciences

The deep learning community is accustomed to open science, as many datasets, models, programming frameworks, and scientific outcomes are publicly released by both academia and companies continuously. However, while deep learning can openly provide state-of-the-art models for old and new problems in neuroscience, theoretical understanding, formalization, and standardization are often yet to be achieved, which may prevent adoption in other research endeavors. From a technical standpoint, deep networks are a viable tool for many tasks involving data from the brain sciences. Image classification has arguably been the task in which deep neural networks have had the highest momentum in terms of pushing the state of the art forward. This translates now into a rich taxonomy of architectures and pre-trained models that consistently maintain interesting performances in pattern recognition across a number of image domains. Pattern recognition is indeed central for diagnostic purposes, in the form of classification of images with pathological features (e.g., types of brain tumors or meningiomas), segmentation of structures (such as the brain, brain tumors, or stroke lesions), classification of signals (e.g., classification of electromyography or electro encephalography data), as well as for action recognition in Human-Computer Interfaces (HCIs) and Brain-Computer Interfaces (BCIs), where the complex systems underlying human behavior and mind must be interpreted, processed and used by artificial systems (see [32] for a larger review of BCIs). The initiatives BRain Tumor Segmentation (BRATS) Challenge (https://www.med.upenn.edu/cbica/brats/) [33], Ischemic Stroke LEsion Segmentation (ISLES) Challenge (https://www.isles-challenge.org/) [34,35], and Ninapro (http://ninaweb.hevs.ch/node/7) [36] are examples of data releases for which above-mentioned tools proved effective. There are models learning image-to-image functions capable of enhancing data, preprocessing it, correcting artifacts and aberrations, allowing smart compression as well as super-resolution, and even expressing cross-modal transformations between different acquisition apparatuses. In the related tasks of object tracking, action recognition, and pose estimation, research results from the automotive sector or crowd analysis have inspired solutions for behavioral neuroscience, especially in animal behavioral studies. When dealing with sequences, deep networks success in computer vision has inspired CNN-based approaches to EEG and EMG studies [37,38], either with or without relying on 2D data, given that mathematical convolution has a 1D version, and 1D signals have 2D spectra. Other architectures more directly instantiate temporal and sequential aspects, e.g., Recurrent Neural Networks (RNNs) such as the Long Short Term Memory (LSTM) [39] and Gated Recurrent Units (GRUs) [40], and they too can be applied to sequence problems and sub-tasks in neuroscience, such as decoding time-dependent brain signals. Although deep neural networks do not explicitly model the nervous system, they are inspired by biological knowledge and mimic some aspects of biological computation and dynamical systems. This has inspired new comparative studies and analogous approaches to learning and perception in a unique way among machine learning algorithms [41]. Many neuroinformatic studies demonstrate how novel deep learning concepts and methods apply to neurological data [12]. However, they often showcase new advanced achievements in performance metrics that do not translate directly to new accepted neuroscience discoveries or clinical best practices.

Such results are very often published together with open code repositories, allowing for reproducibility, yet they may not be explicitly organized for widespread routine adoption in domains different from machine learning. Algorithms are usually written in open programming languages like Python [42], R [43], Julia [44], and deep learning design frameworks such as TensorFlow, PyTorch or Flux [45]. Still, they are more inspiring to the experienced machine learning researcher rather than practically helpful to end users such as neuroscientists. In fact, to successfully build a deep learning application from scratch, vast knowledge is needed in the data science aspect of the task and in coding, as much as in the theoretical and experimental foundations and frontiers of the application domain, here being neuroscience. For the above reasons, the open source and open science domains are promising frames for common development and testing of relevant solutions for neuroscience, as they provide an active flow of ideas and robust diversification, avoiding “reinvention of the wheel”, harmful redundancies, or starting from completely blank states. As a contribution to clarifying the current situation and reducing the workload for researchers, this work collects and analyzes several open libraries that implement and facilitate deep learning applications in neuroscience, with the aim of allowing scientists worldwide to identify the most suitable options for their inquiries and clinical tasks.

3. Materials and Methods

The large corpus of available open code makes it useful to specify what qualifies as a coding library or a framework rather than as a model accompanied by utilities for the present scope. In programming, a library is a collection of pre-coded functions and object definitions, often relying on one another and written to optimize programming for custom tasks. The functions are considered useful and unmodified across multiple unrelated programs and tasks. The main program at hand calls the library in the control flow specified by the end users. A framework is a higher level concept, akin to the library, but typically with pre-designed control flows in which custom code from the end users is inserted.

In this review, a repository that simply collects a set of functions that defines and instantiates a deep learning model is not considered a library. On the contrary, a collection of notebooks that allows us to train, retrain, and test models with different architectures, also taking care of data preprocessing and preparation, fully meets the present scopes. The explicit definition given by the authors, their aims, and their level of maintenance were relevant in determining if a repository would be considered a library (or toolkit/toolbox, etc.). Open code for this review comprises code for proprietary languages such as MATLAB, the reasons being the compatibility with free languages such as GNU Octave (https://octave.org/) (where noted), and the general value of open accessing algorithms. For the sake of the review, several resources were queried or scanned. Google Scholar was queried with:

• Allintitle: “deep learning library”;
• Allintitle: “deep learning toolbox”;
• Allintitle: “deep learning package”;
• “deep learning library|toolbox|package” AND “neuroscience|neuroimaging”;
• “deep learning library|toolbox|package” AND “EEG|EMG”;
• “deep learning library” OR “deep learning toolbox” OR “deep learning package” -“MATLAB deep learning toolbox”

preserving the top 100 search results, ordered for relevance by the engine algorithm. On PubMed the queries were:

• opensource (deep learning) AND (toolbox OR toolkit OR library);
• (EEG OR EMG OR MRI OR (brain (X-ray OR CT OR PT))) (deep learning) AND (toolbox OR toolkit OR library).

Moreover, the site https://open-neuroscience.com/ was scanned specifically for “deep learning” mentions. Stemming citations and automatic recommendations from the engines of the hosting and publishing platforms were also analyzed. The time window was unrestricted in the past, given the recent development of the field, and the search was finished by 22 November 2022 for the selection of library entries. Data regarding library use were updated to 17 April 2023.

The collected libraries were organized according to the principal aim, in the form of data type processed or the supporting function in the workflow, thus dividing:

• Libraries for sequence data (e.g., EMG, EEG)
• Libraries for image data (including scalar volumes, 4-dimensional data as in fMRI, video)
• Libraries and frameworks for further data types and abstractions (including data handling, evaluation, and cloud platforms)

In each category, a set of three tables present separately the results related to the following library characteristics:

• Domain of application
• Model engineering
• Technology and sources

The domain of application comprises the Neuroscience area, the Data types handled, the provision of Datasets, and the machine learning Task to which the library is dedicated. When available (73 entries out of 74), a publication is referenced for the library entry in the domain table. Together with the repositories, referenced publications contain valuable information every potential user should check before experimenting, such as the data sets leveraged and use cases intended by the original authors. The model engineering tables include information on the architecture of Models manageable in the library, the DL (Deep Learning) framework and Programming language main dependencies, and the possibility of Customization for the model structure or training parameters. Technology and sources refer to the type of Interface available for a library, whether it works Online/Offline, specifically with real-time data or logged data. Maintenance refers to the ongoing activity of releasing features, solving issues and bugs, or offering support through channels (considered active with commits or releases in 2022), Source specifies where code files and instructions are made available. Stars (Forks) refers to the counts of “stars” and “forks” of the repositories, by which the number of users and possible new developers could be approximately estimated. Contributors is the number of people adding code and features to the repository, as declared on site (effective contributors might be differently acknowledged), which is useful to estimate the amount of teamwork, developer support, as well as the inclination to customization that could be expected for the given library. The entry “(*)” signals missing data.

4. Results

The analysis of the literature allowed us to select a total of 74 entries for the tables, with publications that describe libraries implementing or empowering deep learning applications for neuroscience. Despite open source and effectiveness, several publications did not provide an ecosystem of reusable functions. Proofs of concept and single-shot experiments were discarded. Please, refer to the Abbreviations section for acronyms from the entire paper and specifically the following tables.

4.1. Libraries for Sequence Data

Libraries and frameworks for sequence data are shown in

Table 1. Domains of applications for the libraries and frameworks processing sequence data.

Name	Neuroscience Area	Data Type	Datasets	Task
braindecode [46]	General	EEG, MEG	External	Classification
DeepEEG	Electrophysiology	EEG	No	Classification
DeLINEATE [47]	General	Images, sequences	External	Classification
DN3 [48]	BCI	EEG	No	Classification
EEG-DL [49]	BCI	EEG	No	Classification
ExBrainable [50]	Electrophysiology	EEG	External	Classification, XAI
gumpy [51]	BCI	EEG, EMG	No	Classification
SANTIA [52]	Electrophysiology	Local Field Potentials	No	Processing
Traja [53]	Behavioural neuroscience	Trajectories	No	Prediction, Classification, Synthesis
VAME [54]	Behavioral neuroscience	Trajectories	No	Embedding, Clustering
VARDNN toolbox [55]	Connectomics (Functional Connectivity)	Sequences (BOLD signal)	No	Time series causal analysis

(domains of application),

Table 2. Model engineering specifications for the libraries and frameworks processing sequence data.

Name	Models	DL Framework	Customization	Programming Language
braindecode	1-D CNN	PyTorch	Yes (weights, model)	Python
DeepEEG	MLP, 1,2,3-D CNN, LSTM	Keras, TensorFlow	Yes (weights)	Python
DeLINEATE	CNN	Keras, TensorFlow	Yes (weights, model)	Python
DN3	MLP	PyTorch	Yes	Python
ExBrainable	CNN	PyTorch	Yes (weights)	Python
EEG-DL	Miscellaneous	TensorFlow	Yes (weights, model)	Python, MATLAB
gumpy	CNN, LSTM	Keras, Theano	Yes (weights, model)	Python
SANTIA	MLP, LSTM, 1-D CNN	Deep Learning Toolbox	Yes (weights, model)	MATLAB
Traja	LSTM, VAE, GAN	PyTorch	Yes (weights, model)	Python
VAME	VAE	PyTorch	Yes (weights, size)	Python
VARDNN toolbox	Vector Auto-Regressive DNN	TensorFlow	Yes (weights)	Python

(models characteristics),

Table 3. Technological aspects and code sources for the libraries and frameworks processing sequence data. The entry “(*)” signals missing data.

Name	Interface	Online/Offline	Maintenance	Source	Stars (Forks)	Contributors
braindecode	CLI	Offline	Active	https://github.com/braindecode/braindecode	463 (123)	26
DeepEEG	Colab Notebooks	Offline	Inactive	https://github.com/kylemath/DeepEEG	213 (55)	2
DeLINEATE	GUI, Colab Notebooks	Offline	Active	https://bitbucket.org/delineate/delineate	(*)	3
DN3	CLI	Offline	Inactive	https://github.com/SPOClab-ca/dn3	50 (16)	4
EEG-DL	CLI	Offline	Active	https://github.com/SuperBruceJia/EEG-DL	640 (179)	1
ExBrainable	GUI	Offline	Active	https://github.com/CECNL/ExBrainable	1 (2)	4
gumpy	CLI	Online, Offline	Inactive	https://github.com/gumpy-bci	61 (22)	4
SANTIA	GUI	Offline	Inactive	https://github.com/IgnacioFabietti/SANTIAtoolbox	4 (2)	1
Traja	CLI	Offline	Active	https://github.com/traja-team/traja	71 (23)	10
VAME	CLI	Offline	Active	https://github.com/LINCellularNeuroscience/VAME	130 (43)	5
VARDNN toolbox	CLI	Offline	Active	https://github.com/takuto-okuno-riken/vardnnpy	2 (0)	2

(technologies and sources). The majority of models process EEG signals, which are among the most common types of sequential data in neuroscience research. A common objective is deducing the activity or state of the subject based on temporal or spectral (2D) patterns. Deep Learning is capable of bypassing some of the preprocessing steps often required by other common statistical and engineering techniques, and it comprises both 1D and 2D approaches through MLPs, CNNs, or RNNs architectures. An example of a sequence-oriented library is gumpy, whose intended area of application is that of BCIs, where decoding a signal is the first step towards communication and interaction with a computer or robotic system. Given the setting, gumpy allows working with EEG or EMG data and suits them with specific defaults, e.g., 1-D CNNs, or LSTMs. Similarly to ExBrainable, it was validated on data from the BCI Competition IV (https://www.bbci.de/competition/iv/).

Notable mentions in the sequence category are the library Traja and the VARDNN toolbox, as they depart from the common scenarios of previous examples. Traja stands out as an example of less usual sequential data, namely trajectory data (sequences of coordinates in 2 or 3 dimensions, through time). Moreover, in Traja sequences are modeled and analyzed employing the advanced architectures of Variational AutoEncoders (VAEs) and Generative Adversarial Networks (GANs), usually encountered in image tasks. With different theoretical backgrounds, both architectures allow simulation and characterization of data through their statistical properties. The VARDNN toolbox enables analyses on blood-oxygen-level-dependent (BOLD) signals in the established domain of functional Magnetic Resonance Imaging (fMRI) but uses a unique approach to autoregressive processes mixed with deep neural networks, allowing to perform causal analysis and to study functional connections between brain regions through their patterns of activity in time. It was developed from the data set ADNI-2 (73 subjects) from the Alzheimer’s Disease Neuroimaging Initiative (https://adni.loni.usc.edu/).

Overall, the libraries oriented to sequence data analysis are mainly directed at the classification of EEG signals, whose variety of acquisition settings and downstream applications could be largely approached with the aid of deep models as a part of the pipeline. Other types of sequence data in neuroscience could be processed by newer or harder-to-retrieve libraries. Despite the fact that preprocessing and domain-specific features require special care, sequence model can still be applied in principle to these data to perform several machine learning tasks on magnetoencephalography (MEG), electrocorticography (ECoG), spike train data, and more. The expert end user may apply or adapt the libraries mentioned above to new domains, or develop new applications, possibly leveraging the open source of available code.

4.2. Libraries for Image Data

Libraries and frameworks for image data are shown in

Table 4. Domains of applications for the libraries and frameworks processing image data.

Name	Neuroscience Area	Data Type	Datasets	Task
Allen Cell Structure Segmenter [57]	Microbiology, Histology	3D-fluorescence microscopy	No	Segmentation
ALMA [58]	Behavioral neuroscience	Video	External	Pose estimation, Classification
ANTsX [59] (ANTsPyNet, ANTsRNet)	Neuroimaging	MRI	No	Classification, Segmentation, Registration, Superresolution
ATLASS [60]	Medical Imaging	Images	No	Annotation, Classification
AxonDeepSeg [61]	Microbiology, Histology	SEM, TEM	External	Segmentation
BART [62]	Medical Imaging	MRI	No	Reconstruction
Brainstorm [63]	Medical Imaging	MRI	No	Synthesis, Augmentation
CASCADE [64]	Electrophys.	2-photon calcium video, sequences	Yes	Event detection
CDeep3M2 [65]	Microbiology, Histology	Microscopy	Yes	Segmentation
CERR [66]	Oncology, Radiomics	Images	No	Segmentation, Outcome prediction
ClinicaDL [67]	Neuroimaging	MRI, PET	External	Classification, Segmentation
DANNCE [68]	Behavioral neuroscience	Video	Yes	Pose estimation
DeepBehavior [69]	Behavioral neuroscience	Video	Yes	Pose estimation
DeepBhvTracking [70]	Behavioral neuroscience	Video	No	Pose estimation
DeepCINAC [71]	Electrophys.	2-photon calcium video	No	Classification
DeepInfer [72]	Medical Imaging	Images (3D)	No	Classification, Segmentation
DeepLabCut [73]	Behavioral neuroscience	Video	No	Pose estimation
DeepLabStream [74]	Behavioral neuroscience	Video	No	Pose estimation
DeepNeuro [75]	Neuroimaging	Images (fMRI, miscellaneous)	No	Classification, Segmentation, Synthesis
DeepNeuron [76]	Morphology	Images (2D, 3D)	No	Classification, Segmentation
DeepPoseKit [77]	Behavioral neuroscience	Video	No	Pose estimation
DeLINEATE [47]	Medical Imaging	Images, sequences	External	Classification
DeepVOG [78]	Oculography	Images, Video	Demo	Segmentation
DLTK [79]	Medical Imaging	Images	No	Classification, Segmentation
DNNBrain [80]	Brain mapping	Images	No	Classification
FastSurfer [81]	Neuroimaging	MRI	No	Segmentation
fetal-code [82]	Neuroimaging	rs-fMRI	External	Segmentation
GaNDLF [83]	Medical Imaging	Images (2D, 3D)	External	Segmentation, Regression, XAI
hipotalamus_seg [84]	Neuroimaging	MRI	No	Segmentation
ivadomed [85]	Neuroimaging	Images (2D, 3D)	No	Classification, Segmentation
LEAP [86], SLEAP [87]	Behavioral neuroscience	Video	No	Pose estimation
MARS, BENTO [88]	Behavioral neuroscience	Video	Yes	Pose estimation, Classification, Action recognition, Tag
MesoNet [89]	Neuroimaging	Images (fluoresc. microscopy)	External	Segmentation, Registration
MEYE [90]	Oculography	Images, Video	Yes	Segmentation
MIScnn [91]	Medical Imaging	Images (2D, 3D)	No	Segmentation
Neurite, Neuron [92]	Neuroimaging	Images	No	Segmentation
NiftyNet [93]	Medical Imaging	MRI, CT	No	Classification, Segmentation, Synthesis
NiftyTorch [94]	Neuroimaging	Images (2D, 3D)	No	Classification, Segmentation, Synthesis
nnU-Net [95]	Medical Imaging	Images (2D, 3D)	No	Segmentation
PyTC [96]	Connectomics	Images (2D, 3D)	No	Segmentation
ScLimbic [97]	Neuroimaging	MRI	External	Segmentation
SimBA [98]	Behavioral neuroscience	Video	No	Pose estimation
SynthStrip [99]	Neuroimaging	Images (3D)	No	Segmentation, Extraction
VesicleSeg [100]	Microbiology, Histology	EM	No	Segmentation
Visual Fields Analysis [101]	Eye tracking, Behavioral neuroscience	Video	No	Pose estimation, Classification
Volume Segmantics [102]	Neuroimaging	Images (3D)	No	Segmentation
VoxelMorph [103], HyperMorph [104]	Neuroimaging	MRI	No	Registration

(domains of application),

Table 5. Model engineering specifications for the libraries and frameworks processing image data.

Name	Models	DL Framework	Customization	Programming Language
Allen Cell Structure Segmenter	CAE	PyTorch	No	Python
ALMA	CNN	Unspecified	No	Python
ANTsX (ANTsPyNet, ANTsRNet)	CNN, CAE, GAN	Keras, TensorFlow	Yes	Python, R, C++
ATLASS	CNN	FastAI	Yes	Python
AxonDeepSeg	CAE	TensorFlow	Yes (weights)	Python
BART	CAE, VAE	BART, TensorFlow	Yes	Python
Brainstorm	3D-CAE	Keras, TensorFlow	Yes (weights)	Python
CASCADE	1-D CNN	TensorFlow	Yes (weights)	Python
CDeep3M2	CAE	TensorFlow	Yes (weights)	Python
CERR	CAE	CERR	Yes	Octave, MATLAB
ClinicaDL	CNN, CAE	PyTorch	Yes	Python
DANNCE	3D-CNN	PyTorch	Yes (weights)	Python, MATLAB
DeepBehavior	CNN	TensorFlow	Yes (weights)	Python, MATLAB
DeepBhvTracking	CNN	TensorFlow	Yes (weights)	Python, MATLAB
DeepCINAC	DeepCINAC (CNN+LSTM)	Keras, TensorFlow	Yes (weights)	Python
DeepInfer	3D-CNN, CAE	Unspecified	Yes (weights)	Python, C++ (3D Slicer)
DeepLabCut	CNN	TensorFlow	Yes (weights)	Python
DeepLabStream	CNN	TensorFlow	Yes (weights)	Python
DeepNeuro	CNN, CAE, GAN	Keras, TensorFlow	Yes (weights, model)	Python
DeepNeuron	2,3-D CNN	Unspecified (Vaa3D)	Yes (weights)	C++
DeepPoseKit	CNN	Keras, TensorFlow	Yes (weights)	Python
DeLINEATE	CNN	Keras, TensorFlow	Yes (weights, model)	Python
DeepVOG	CAE	TensorFlow	No	Python
DLTK	CNN, CAE	Tensorflow	Yes (weights)	Python
DNNBrain	CNN	PyTorch	Yes (model)	Python
FastSurfer	CNN	PyTorch	Yes (weights)	Python (FreeSurfer)
fetal-code	2-D CNN	TensorFlow	No	Python
GaNDLF	CNN, CAE	PyTorch	Yes	Python
hipotalamus_seg	3D-CNN	Keras, TensorFlow	Yes (weights)	Python
ivadomed	2,3-D CNN, CAE	PyTorch	Yes (weights, model)	Python
LEAP, SLEAP	CNN, CAE	TensorFlow	Yes (weights, model)	Python
MARS, BENTO	CNN	TensorFlow	Yes (weights)	Python
MesoNet	CNN, CAE	Keras, TensorFlow	No	Python
MEYE	CAE, CNN	TensorFlow	Yes (model)	Python
MIScnn	2,3-D CNN	TensorFlow	Yes (weights)	Python
Neurite, Neuron	VAE	Keras, TensorFlow	Yes (weights)	Python
NiftyNet	CNN	TensorFlow	Yes	Python
NiftyTorch	CNN, CAE, GAN	PyTorch	Yes	Python
nnU-Net	2,3-D CAE	PyTorch	Yes	Python
PyTC	2,3-D CAE	PyTorch	Yes	Python
ScLimibic	3-D CAE	Neurite, TensorFlow	No	Python (FreeSurfer)
SimBA	CNN	TensorFlow	Yes (weights)	Python
SynthStrip	3-D CAE	PyTorch	No	Python
VesicleSeg	CNN	PyTorch	No	Python
Visual Fields Analysis	DeepLabCut	TensorFlow, DeepLabCut	Yes (weights)	Python
Volume Segmantics	CAE	PyTorch	Yes (weights)	Python
VoxelMorph, HyperMorph	CAE	TensorFlow	Yes (weights)	Python

(models characteristics),

Table 6. Technological aspects and code sources for the libraries and frameworks processing image data. The entry “(*)” signals missing data.

Name	Interface	Online/Offline	Maintenance	Source	Stars (Forks)	Contributors
Allen Cell Structure Segmenter	GUI, Jupyter Notebooks	Offline	Active	https://github.com/AllenCell/aics-ml-segmentation	24 (4)	2
ALMA	GUI	Offline	Active	https://github.com/sollan/alma	7 (1)	2
ANTsX (ANTsPyNet, ANTsRNet)	CLI	Offline	Active	https://github.com/ANTsX	961 (354)	55
ATLASS	Jupyter Notebooks	Offline	Inactive	https://github.com/adines/ATLASS	2 (1)	2
AxonDeepSeg	Jupyter Notebooks	Offline	Active	https://github.com/axondeepseg/axondeepseg	106 (28)	21
BART	CLI	Offline	Active	https://github.com/mrirecon/deep-deep-learning-with-bart	4 (0)	1
Brainstorm	CLI	Offline	Inactive	https://github.com/xamyzhao/brainstorm	387 (93)	1
CASCADE	GUI, Colab Notebooks	Offline	Active	https://github.com/HelmchenLabSoftware/Cascade	74 (26)	7
CDeep3M2	GUI, Colab Notebooks	Offline	Active	https://github.com/CRBS/cdeep3m2	4 (2)	2
CERR	GUI	Offline	Active	https://github.com/cerr/CERR	170 (95)	7
ClinicaDL	GUI, Colab Notebooks	Offline	Active	https://github.com/aramis-lab/clinicadl
DANNCE	GUI	Offline	Inactive	https://github.com/spoonsso/dannce/	162 (23)	7
DeepBehavior	GUI	Offline	Inactive	https://github.com/aarac/DeepBehavior	29 (17)	1
DeepBhvTracking	GUI	Offline	Inactive	https://github.com/SunGL001/DeepBhvTracking	3 (0)	1
DeepCINAC	GUI, Colab Notebooks	Offline	Active	https://gitlab.com/cossartlab/deepcinac	7 (0)	10
DeepInfer	GUI	Offline	Active	http://www.deepinfer.org/	24 (14)	4
DeepLabCut	GUI, Colab Notebooks	Offline	Active	https://github.com/DeepLabCut/DeepLabCut	3600 (1500)	102
DeepLabStream	GUI	Online	Active	https://github.com/SchwarzNeuroconLab/DeepLabStream	45 (8)	5
DeepNeuro	CLI	Offline	Active	https://github.com/QTIM-Lab/DeepNeuro	113 (35)	3
DeepNeuron	GUI	Offline	Inactive	https://github.com/Vaa3D/vaa3d_tools/tree/master/hackathon/MK/DeepNeuron	92 (69)	69
DeepPoseKit	GUI	Offline	Inactive	https://github.com/jgraving/DeepPoseKit	353 (83)	6
DeLINEATE	GUI, Colab Notebooks	Offline	Active	bitbucket.org/delineate/delineate	(*)	3
DeepVOG	CLI	Offline	Inactive	https://github.com/pydsgz/DeepVOG	131 (58)	4
DLTK	CLI	Offline	Inactive	https://github.com/DLTK/DLTK	1400 (408)	6
DNNBrain	CLI	Offline	Active	https://github.com/BNUCNL/dnnbrain	37 (39)	10
FastSurfer	CLI	Offline	Active	https://github.com/Deep-MI/FastSurfer	338 (83)	12
fetal-code	GUI, Colab Notebooks	Offline	Active	https://github.com/saigerutherford/fetal-code	12 (5)	2
GaNDLF	GUI	Offline	Active	https://github.com/CBICA/GaNDLF	85 (53)	30
hipotalamus_seg	CLI	Offline	Active	https://github.com/BBillot/hypothalamus_seg	18 (5)	1
ivadomed	CLI	Offline	Active	https://github.com/ivadomed/ivadomed	146 (151)	34
LEAP, SLEAP	CLI, GUI, Colab Notebooks	Online	Active	https://github.com/talmolab/sleap	278 (61)	26
MARS, BENTO	GUI, MATLAB GUI, Jupyter Notebooks	Offline	Active	https://github.com/neuroethology	37 (8)	3
MesoNet	GUI, Colab Notebooks	Offline	Active	osf.io/svztu	(*)	3
MEYE	Web app	Online, Offline	Active	pupillometry.it	21 (5)	2
MIScnn	Jupyter Notebook	Offline	Inactive	https://github.com/frankkramer-lab/MIScnn	357 (116)	6
Neurite, Neuron	CLI	Offline	Active	https://github.com/adalca/neurite	279 (59)	11
NiftyNet	CLI	Offline	Inactive	https://github.com/NifTK/NiftyNet	1300 (408)	41
NiftyTorch	CLI	Offline	Active	https://github.com/NiftyTorch/NiftyTorch.doc	34 (8)	3
nnU-Net	CLI	Offline	Active	https://github.com/MIC-DKFZ/nnUNet	3500 (1200)	38
PyTC	CLI	Offline	Active	https://github.com/zudi-lin/pytorch_connectomics	139 (67)	28
ScLimbic	CLI	Offline	Active	https://surfer.nmr.mgh.harvard.edu/fswiki/ScLimbic	(*)	(*)
SimBA	GUI	Offline	Active	https://github.com/sgoldenlab/simba	201 (115)	9
SynthStrip	CLI	Offline	Active	https://github.com/freesurfer/freesurfer/tree/dev/mri_synthstrip	(*)	(*)
VesicleSeg	GUI	Offline	Active	https://github.com/Imbrosci/synaptic-vesicles-detection	4 (3)	2
Visual Fields Analysis	GUI	Offline	Active	https://github.com/mathjoss/VisualFieldsAnalysis	1 (3)	4
Volume Segmantics	CLI, API	Offline	Active	https://github.com/DiamondLightSource/volume-segmantics	7 (3)	1
VoxelMorph, HyperMorph	CLI	Offline	Active	https://github.com/voxelmorph/voxelmorph	1800 (534)	13

(technologies and sources). Computer vision and 2D image processing are arguably the fields in which deep learning has achieved the most impressive and state-of-art defining results, often inspiring and translating breakthroughs in other domains. Classification and segmentation (i.e., the separation of parts of the image based on their classes) are the most common tasks addressed by image processing libraries. Magnetic resonance is the primary source of data; however, various deep learning libraries are built for microscopic and eye-tracking data as well. Most of the libraries collected in our analysis took advantage of classical CNN architectures for classification, Convolutional AutoEncoders (CAEs) for segmentation, and GANs for synthesis. It is common to employ transfer learning to lessen the computational and memory burden during the training phase and take advantage of pre-trained models. Transfer learning consists of initializing models with parameters learned on usually larger data sets, possibly from different domains and tasks, with varying amounts of further training in the target domain. The best such examples are pose-estimation libraries extending the DeepLabCut system, arguably the most relevant project on the topic. DeepLabCut is an interactive framework for labeling, training, testing, and refining models that originally exploits the weights learned from ResNets (or newer architectures) on the ImageNet data. The results match human annotation using quite a few training samples, holding for many (human and non-human) animals and settings. Validated data sets comprise TRI-MOUSE (161 data points), Parenting Mouse (542), MARMOSET (7600), FISH (100), and HORSE (8114). The documentation, demonstrative notebooks, and tools offered by the Mathis Lab allow different levels of understanding and customization of the process with high levels of robustness. Among the considered libraries, two set apart from the majority given the type of tasks they perform: GaNDLF addresses eXplainable AI (XAI), i.e., Artificial Intelligence whose decisions and outputs can be understood by humans through more transparent mental models; ANTsX performs both the co-registration step and super-resolution as a quality enhancing step for neuroimages, with the former being usually performed by traditional algorithms. GaNDLF sets its goal as the provision of deep learning resources in different layers of abstraction, allowing medical researchers with virtually no ML knowledge to perform robust experiments with models trained on carefully split data, with augmentations and preprocessing, under standardized protocols that can easily integrate interpretability tools such as Grad-CAM [56] and attention maps, which highlight the parts of an image according to how they influenced a model outcome. It was validated on 7 data sets comprising from 371 up to 180,000 images of different systems ranging from brain MRI to dental X-ray and eye fundus. The ANTsX ecosystem is of similar wide scope and is intended to build workflows on quantitative biology and medical imaging data, both in Python and R languages. Packages from the same ecosystem perform registration of brain structures (by classical methods) as well as brain extraction by deep networks, aggregating structural MRI data sets for over 1200 subjects.

4.3. Libraries for Further Data Types and Abstractions

Libraries for further data types and abstractions are shown in

Table 7. Domains of applications for the libraries and frameworks for special applications.

Name	Neuroscience Area	Data Type	Datasets	Task
DANCE [106]	Single cell analysis	Gene sequences	External availability	Clustering, Classification, Prediction
PsychRNN [107]	Computational neuroscience	Sequences	No	Classification, Prediction, Cognitive tasks
PyCog [108]	Computational neuroscience	Sequences	No	Classification, Prediction, Cognitive tasks
THINGvision [109]	Computational neuroscience	Images, Text	External availability	Classification
TorchDIVA [110]	Speech production	Sequences	No	Audio synthesis
COINSTAC [111]	Neuroimaging	Img	No	Federated learning, Classification, Segmentation
Fed-BioMed [112]	Neuroimaging	Img	No	Federated learning, Classification, Aggregation
FeTS [113]	Neuroimaging	Img	No	Federated learning, Segmentation
MeDaS [114]	Neuroimaging	Img	No	Utilities, Classification, Segmentation, Object detection
MONAI [115]	General	General	External availability	General
NeuroCAAS [116]	General	General	External availability	General
NeuroGym [117]	Behavioral, cognitive neuroscience	General	Internal	Behavioral, cognitive task generation, evaluation
NiftyNet [93]	Neuroimaging	Img	No	Utilities, Classification, Segmentation, Regression, Synthesis
OpenFL [118]	General	General	No	Federated learning, Segmentation
pymia [119]	General	Img	No	Utilities (data handling, evaluations)
TorchIO [120]	Imaging	All images	No	Augmentation

(domains of application),

Table 8. Model engineering specifications for the libraries and frameworks for special applications.

Name	Models	DL Framework	Customization	Programming Language
DANCE	GNN	PyTorch	Yes	Python
PsychRNN	RNN	Tensorflow	Yes	Python
PyCog	RNN	Theano	Yes	Python
THINGvision	CNN, RNN, Transformers	PyTorch, TensorFlow	No	Python
TorchDIVA	CNN	PyTorch	Yes (weights)	Python
COINCSTAC	CNN	COINSTAC	Yes	JavaScript, Python
Fed-BioMed	VAE	PyTorch	Yes (weights)	Python
FeTS	3D-ResUNet	PyTorch	Yes (weights)	Python
MeDaS	2,3-D CNN, CAE	PyTorch, TensorFlow	Yes	Python
MONAI	General	PyTorch	Yes	Python
NeuroCAAS	CNN	TensorFlow	Yes	Python
NeuroGym	RNN, General	Keras, TensorFlow, PyTorch	Yes	Python
NiftyNet	CNN, CAE, GAN	TensorFlow	Yes	Python
OpenFL	General	Keras, TensorFlow, PyTorch	Yes	Python
pymia	General	General	Yes	Python
TorchIO	CNN	PyTorch	Yes	Python

(models characteristics),

Table 9. Technological aspects and code sources for the libraries and frameworks for special applications. The entry “(*)” signals missing data.

Name	Interface	Online/Offline	Maintenance	Source	Stars (Forks)	Contributors
DANCE	CLI	Offline	Active	https://github.com/OmicsML/dance	206 (15)	8
PsychRNN	None	Offline	Active	https://github.com/murraylab/PsychRNN	122 (38)	8
PyCog	None	Offline	Inactive	https://github.com/xjwanglab/pycog	45 (29)	1
THINGvision	None	Offline	Active	github.com/ViCCo-Group/THINGSvision	108 (17)	12
TorchDIVA	CLI	Offline	Active	https://github.com/skinahan/DIVA_PyTorch	12 (0)	1
COINSTAC	GUI	Offline	Active	https://github.com/trendscenter/coinstac	35 (19)	19
Fed-BioMed	CLI	Offline	Active	https://gitlab.inria.fr/fedbiomed	6 (0)	26
FeTS	GUI, CLI	Offline	Active	https://github.com/FETS-AI/Front-End	55 (6)	2
MeDaS	GUI	Offline	Inactive	https://medas.bnc.org.cn/	(*)	(*)
MONAI	GUI, Colab Notebooks	Offline	Active	github.com/Project-MONAI/MONAI	4000 (776)	151
NeuroCAAS	GUI, Jupyter Notebooks	Offline	Active	github.com/cunningham-lab/neurocaas	25 (22)	6
NeuroGym	Colab Notebooks	Offline	Active	https://neurogym.github.io	(*)	4
NiftyNet	CLI	Offline	Inactive	https://github.com/NifTK/NiftyNet	1300 (408)	41
OpenFL	CLI	Offline	Active	https://github.com/securefederatedai/openfl	484 (134)	49
pymia	CLI	Offline	Active	https://github.com/rundherum/pymia	54 (12)	4
TorchIO	GUI, CLI	Offline	Active	torchio.rtfd.io	1700 (204)	49

(technologies and sources). There are two ways libraries listed in this section differ from the previous ones. On the one hand, they can process data types that do not fall into the sequence or image category. On the other hand, they are the product of projects that transcend the training of deep learning models, i.e., larger scope frameworks, deep learning support functions (e.g., preprocessing and data augmentations pipelines), and services or infrastructure to host computational experiments. NeuroCAAS is an ambitious project that both standardizes experimental schedules and analyses and offers computational resources on the cloud. The platform lifts the burden of configuring and deploying data analysis tools, also guaranteeing replicability and readily available usage of pre-made pipelines with high efficiency. Other platform-oriented libraries are concerned with federated learning, i.e., the training of deep learning models on separated and private datasets, a relevant issue in healthcare. MONAI is a project that brings deep learning tools to many health and biology problems. The paradigm builds on PyTorch and aims at unifying healthcare AI practices throughout both academia and enterprise research, not only in the model development but also in the creation of shared annotated datasets. It also focuses on deployment and work in real-world clinical production, settling as a strong candidate for being the standard solution in the domain. Importantly, it is a commonly used framework for the 3D variations of UNet [105] lately dominating the yearly BraTS challenge [33] (see at http://braintumorsegmentation.org/). In this regard, nnU-Net is a narrower scope framework explicitly for building UNet-like models, focused on data-driven self-configuration of training hyperparameters, reducing the burden on researchers and practitioners. It was validated on 23 public data sets of biomedical interest with great success. PsychRNN, PyCog and THINGvision, as well as NeuroGym are libraries that bridge deep learning research and computational neuroscience. They are concerned with studying how artificial neural systems solve the same tasks that animal and human brains are subject to. The aims are those of simulating, modeling, learning representations, and reverse engineering cognition and behavior.

5. Discussion

The use of deep learning in neuroscience research requires us to frame a study, or a part of it, as a machine learning problem or task. The setting may or may not require researchers to develop a novel learning architecture and algorithm, train an existing model on the data, or directly apply a trained model to the scientific use case. These situations require different levels of machine learning knowledge and data literacy. Consequently, different libraries and frameworks would be of use. In general, researchers should be aware of what mathematical objects and data objects can represent their experimental data: deep learning typically processes vectors, matrices, and tensors, which could represent signals, images, scalar and tensor fields, and more. Scientists should also consider the nature of data used to train models and how similar they are to use case data, hence how the model would perform on the latter. Importantly, there must be a specific task the model would perform in processing input data and providing the output, e.g., many problems are formulated as classification tasks, while others are very specific to the neuroscience domain. Deep learning models may retain general information from the training data but generally cannot apply it to different tasks without design adaptations. The application of deep learning to neuroscience is thus challenging. Working in teams with different scientific backgrounds is a possible solution for the problem of specialized expertise that deep learning requires. Leveraging the platforms and practices of open science and open source communities offers the support that a working team would still need. Deep learning has the potential hindrance of thriving on large data sets and powerful hardware, but transfer learning and the availability of “model zoos” in many libraries allow researchers to build on systems with great knowledge or efficacy from the start. These systems are not an end in themselves and can be integrated into larger workflows, allowing domain expertise and researchers’ creativity to be enhancing and enhanced.

The tables in this work help to evaluate candidate tools for neuroscience research problems, providing information on the type and domain of data they process, the machine learning task they perform, and whether models are trainable, customizable, or frozen and ready for use.

The panorama of open-source libraries dedicated to deep learning applications in neuroscience is quite rich and diversified. There is a corpus of organized packages that integrate preprocessing, training, testing, and performance analyses of deep neural networks for neurological research. Most of these projects are tuned to specific data modalities and formats, but some libraries are quite versatile and customizable, and there are projects that encompass quantitative biology and medical analysis as a whole. There is a common tendency to develop GUIs, enhancing the user-friendliness of toolkits for non-programmers and researchers unacquainted with the command line interfaces, for example. Visualizations of building blocks, operations, and results allow us to focus on them without further allocating cognitive and time resources to producing the respective codes. Moreover, for the many libraries developed in Python, the (Jupyter) Notebook format appears as a widespread tool both for tutorials, documentation and as an interface to cloud computational resources (e.g., Google Colab [121]). Through notebooks, experimental templates can be modified and adapted in complete environments, where text instructions, dynamic visualization, and code are blended efficiently, can be shared and reproduced, Although learning curves depend on subjective experiences, the availability of such interfaces and instruments reduces the burden on the end user, enhancing accessibility. Apart from specific papers and documentation, and outside of deep learning per se, it is important to make researchers and developers aware of the main topics and initiatives in open culture and neuroinformatics in order to sustain the development of the field. For this reason, the interested reader is invited to rely on competent institutions (e.g., INCF) and databases of open resources (e.g., open-neuroscience) dedicated to neuroscience. Among the possibly missing technologies, the queries employed did not retrieve results in Natural Language Processing libraries dedicated to neuroscience, nor toolkits specifically employing Graph Neural Networks (GNNs), although available in EEG-DL. NLP is actually fundamental in healthcare since medical reports often come in non-standardized forms. Large Language Models (LLMs), Named Entity Recognition (NER) systems and text mining approaches in biomedical research exist [122,123]. GNNs comprise recent architectures that are extremely promising in a variety of fields [124], including biomedical research and particularly neuroscience [125,126]. Even if promising, their application is still less mature than that of computer vision models or time series analysis.

Future Directions

Overall, current libraries cover a wide range of applications, but it seems unlikely that a single deep learning framework could dominate the entire neuroscience field in the near future. Nonetheless, projects such as the PyTorch-based MONAI are strong candidates in unifying ecosystems for deep learning in medicine and biology. DeepLabCut and nnU-Net are also worth mentioning since they are widely applied as blueprints for newer applications, respectively, in pose estimation and segmentation. Investing efforts in open tools and data, accessible documentation, and modularity may guarantee useful results regardless of the specific application field and are indeed foundational to transferring successful tools across domains. Such effort should be paired with data literacy and basic knowledge of machine learning best practices (e.g., how to avoid data leakage between train and test sets, how to assess model performance), paired with digital frameworks with strong priors towards these experimental practices, such as NeuroCAAS. Interpretability and explainability of models depend not only on the researchers’ theoretical understanding but also on transparent models and effective tools to open and shed light on black boxes. In fact, XAI is essential for systems that support decision-making in healthcare or experiments in biomedical sciences. Tools and libraries such as GaNDLF can be expected to gain value. Another relevant aspect of biomedicine is that of privacy and sensitive data. Records cannot be processed anywhere; personal data cannot be published anyhow. Standards and protocols to protect people’s privacy and machine learning models and frameworks that respect them, such as federated learning, are expected to gain momentum. All developments should hold against the intrinsic multimodality of data from the field and the multidisciplinarity required to analyze them. In the end, the interplay of common practices and flexible models can be expected to be of central importance.

6. Conclusions

Although a large and growing number of repositories offer code to build specific models, as published in experimental papers, these resources seldom aim to constitute proper libraries or frameworks for research or clinical practice. Both deep learning and neuroscience gain much value even from sophisticated proofs of concept. In parallel, organized packages are spreading and starting to provide and integrate pre-processing, training, testing, and performance analyses of deep neural networks for neurological and biomedical research. This paper has offered both a historical and a technical context for the use of deep neural networks in neuroinformatics, focusing on open-source tools that scientists can comprehend and adapt to their necessities. At the same time, this work underlines the value of the open culture and points to relevant institutions and platforms for neuroscientists. Although the aim is not restricted to making clinicians develop their own deep models without coding or machine learning background, as was the case in [127], the overall effect of these libraries and sources is to democratize deep learning applications and results, as well as standardize such complex and varied models, supporting the research community in obtaining a proper means to an end and in envisioning then realizing collectively new projects and tools.