Review of Multimodal Data Acquisition Approaches for Brain–Computer Interfaces

Abstract

There have been multiple technological advancements that promise to gradually enable devices to measure and record signals with high resolution and accuracy in the domain of brain–computer interfaces (BCIs). Multimodal BCIs have been able to gain significant traction given their potential to enhance signal processing by integrating different recording modalities. In this review, we explore the integration of multiple neuroimaging and neurophysiological modalities, including electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), electrocorticography (ECoG), and single-unit activity (SUA). This multimodal approach leverages the high temporal resolution of EEG and MEG with the spatial precision of fMRI, the invasive yet precise nature of ECoG, and the single-neuron specificity provided by SUA. The paper highlights the advantages of integrating multiple modalities, such as increased accuracy and reliability, and discusses the challenges and limitations of multimodal integration. Furthermore, we explain the data acquisition approaches for each of these modalities. We also demonstrate various software programs that help in extracting, cleaning, and refining the data. We conclude this paper with a discussion on the available literature, highlighting recent advances, challenges, and future directions for each of these modalities.

1. Introduction

Brain–computer interfaces (BCIs) have emerged as transformative systems that establish a direct communication link between the human brain and external devices, such as computers or prosthetic limbs. By harnessing the power of neural activity, BCIs empower users to control these devices through their cognitive intentions, obviating the necessity for conventional input devices [1]. The applications enable a wide spectrum, ranging from assisting individuals afflicted with paralysis or locked-in syndrome in order to regain communication and control over their surroundings to enabling users to manipulate intricate machinery or engage in video games using cognitive intent. However, despite their immense potential, BCIs remain in the nascent stages of development, beset by numerous technical and ethical challenges that demand scrutiny and resolution before widespread integration [2]. This innovation holds the promise to revolutionize the interface between humans and machines, particularly for individuals whose mobility or speech has been impaired due to injuries or disabilities.

The functioning of BCIs hinges upon the ability to capture and interpret the brain’s intricate electrical signals. These signals arise from the brain’s dynamic activity and serve as a reflection of its cognitive processes [3]. Several methodologies have been employed to measure and decode these signals, including electroencephalography (EEG), which records electrical activity from the scalp, and functional magnetic resonance imaging (fMRI), which captures changes in cerebral blood flow. Additionally, BCIs can harness other modalities, such as functional near-infrared spectroscopy (fNIRS) and intracortical recordings, each contributing unique insights into neural activity.

This review explores the swiftly evolving landscape of BCIs, delving into their diverse modalities and the burgeoning possibilities that the implementation of multi-modalities has to offer. While each modality boasts distinct advantages, they also bear inherent limitations in terms of resolution, spatial specificity, user invasiveness, robustness, and generalization. The integration of multiple neuroimaging modalities, such as EEG, MEG, ECoG, and fMRI, has been proposed to overcome the limitations of single-modality systems [4]. The fusion of multiple modalities presents a solution to surmount these limitations, rendering BCIs more precise, dependable, and robust. For instance, EEG excels with high temporal resolution and non-invasiveness, albeit at the cost of spatial resolution, whereas fMRI affords superior spatial resolution but struggles with temporal precision. Integrating both modalities can bridge these gaps, harnessing their individual strengths synergistically.

The overarching objective of this review is to provide an extensive exploration of the integration of EEG/MEG + fMRI/fNIRS, ECoG + MEG/EEG, MUA/SUA + MEA/ECoG, and fMRI + ECoG/MEA modality combinations within the realm of BCIs. This exploration will encompass an evaluation of the strengths and limitations of each modality in isolation [5], while delving into the transformative potential unlocked by their multimodal fusion. The review aims to illuminate the trajectory of research in this domain, delineating the challenges and opportunities, and thereby contributing to the ongoing advancement of BCIs. Through an in-depth analysis of the modalities, their integration, and their applications, this review endeavors to provide comprehensive insights into the present and future landscape of advanced BCIs.

2. BCI Modalities Under Consideration

There have been multiple technological advancements that promise to gradually enable devices to measure and record signals with high resolution and accuracy in the domain of brain–computer interfaces (BCI) that have been utilized for building working models that can be used in various functionalities. Such utilities are as varied as academic endeavors to commercial memory storage devices currently initiated by Neuralink [6] to understand and abet in curing diseases and disabilities of the brain. We provide a review of various BCI modalities that provide an understanding of their working mechanisms. Subsequently, we embark upon exploring the data acquisition techniques for each of those modalities along with the limitations that each of the standalone setups pose.

2.1. Microelectrode Array (MEA)

Microelectrode array techniques, or MEAs, are a class of invasive brain–computer interfaces that utilize small, multi-electrode arrays to record the activities of individual neurons in the brain. MEA techniques provide a high spatial resolution and temporal precision, allowing for precise monitoring of neural activity at the single-cell level. MEA techniques use micro-fabricated arrays of electrodes that are capable of recording extracellular electrical signals from neurons in the brain. These electrodes are typically made of metal or conductive polymer and are very small, with diameters ranging from a few micrometers to a few hundred micrometers. The electrodes are placed in specific regions of the brain, such as the cortex, where they can record the activities of individual neurons or small groups of neurons.

MEA techniques offer several advantages over other invasive BCI methods, such as intracortical microelectrode arrays. MEA systems are less invasive, as the electrodes are smaller and can be inserted through smaller openings in the skull. Additionally, MEA systems can easily be fabricated using microfabrication techniques, making them more accessible and cost effective than other invasive BCI methods. MEA systems have a wide range of potential applications, including the development of prosthetic devices that can restore motor function in individuals with paralysis. MEA techniques have also been used to study the basic mechanisms of neural function and to develop new treatments for neurological disorders such as epilepsy and Parkinson’s disease. Figure 1 displays MEA implants. The MEA implant is situated within the cortical region and comprises a configuration of 60 electrodes, arranged systematically to facilitate precise neuronal interfacing.

Neuronal growth on the MEA surface is observed, highlighting the array’s integration with neural tissue. Detailed imaging through Scanning Electron Microscopy (SEM) provides structural insights, revealing both the standard MEA and its nano-scale variant, offering a closer look at the fine architecture and surface characteristics essential for effective neuroelectronic interactions. A typical MEA biosensor has a sensory area of 700 μm squared with a length of 5 mm. Within this area, an alignment of 60 electrodes is designed in an 8 × 8 grid. The electrodes are usually placed across with a separation of either 100, 200, or 500 μm of planar TiN [8]. All electrodes have dual usability both for recording and for stimulation. Within the context of either cell or slice cultures, standard coatings are mandatorily applied on MEAs prior to their use for better cellular fixation and extension. Spike signals might be sensed at separations of up to 100 μm from a neuron. Ideally, sites for signal emission are found in an area of 30 μm² around the electrode. Closer proximities guarantee greater signal reception, and spatial resolution is inversely proportional to the signal units picked up by a single electrode, and that translates to less work towards spike sorting. Multi-channel systems enable MEAs that, in turn, enable higher spatial resolutions. High-density MEAs are another class of MEAs that have electrodes with a diameter of only 10 μm housed within a space of only 30 μm apart. There has been an inherent challenge in the processing of extra tiny electrodes while maintaining low levels of impedance and noise simultaneously. This impediment has been eradicated by the incorporation of TiN as the compound for manufacturing the electrodes.

Data Acquisition Approaches and Limitations with a Standalone Setup

Data acquisition in standalone MEA systems is a critical process that involves capturing and processing neural signals directly from the electrodes implanted in the brain. In a standalone setup, the MEA operates independently, typically interfaced with a data acquisition system that amplifies, filters, and digitizes the recorded signals for further analysis. One of the primary advantages of this setup is its capacity for real-time monitoring and data processing, which is essential for applications such as BCIs and neurological research [8].

However, there are some limitations to MEA techniques. One of the main challenges is the low signal-to-noise ratio of the recorded signals, which can make it difficult to distinguish individual neurons from the background noise. Additionally, the small size of the electrodes can make it difficult to maintain stable recordings over long periods of time, as the electrodes can shift or become damaged.

A significant challenge is the low signal-to-noise ratio (SNR) inherent in the recordings [9]. The small size of the electrodes, while beneficial for reducing invasiveness and increasing spatial resolution, can result in weak signals that are easily masked by background noise. This low SNR complicates the isolation of individual neuronal spikes, which is crucial for accurate neural decoding [10]. Additionally, the standalone nature of these systems means they are often constrained by power and processing capabilities, limiting their ability to handle large volumes of data or to perform complex real-time analyses [11].

Another limitation is the long-term stability of the recordings. The electrodes, being in direct contact with the neural tissue, are susceptible to degradation over time due to biological responses such as gliosis, where glial cells proliferate around the electrode site, potentially leading to signal attenuation or loss [12]. Furthermore, electrode drift, caused by slight movements or shifts of the electrodes within the brain tissue, can disrupt the consistency of the recorded signals, making it difficult to maintain reliable long-term data acquisition [13]. These challenges necessitate ongoing research into materials and techniques that can enhance the durability and stability of MEA systems, as well as improvements in signal-processing algorithms to better handle the inherent noise and variability in neural recordings [14].

2.2. Electrocorticography (ECoG)

Electrocorticography (ECoG) is a technique for recording the electrical activity of the brain. ECoG involves placing a grid of electrodes directly on the surface of the brain, underneath the skull, and over the cortical tissue. This technique provides high spatial and temporal resolutions, making it a valuable tool for BCI research.

ECoG has been used to decode motor and speech-related signals for BCI applications. The technique has been successful in decoding the movements of individual fingers and the intention to perform specific movements. ECoG can also be used to decode speech production and comprehension, with applications in speech rehabilitation for patients with speech disabilities. The capability to obtain precise and detailed neural data enables researchers to explore the neural mechanisms underlying complex motor and cognitive tasks, advancing our understanding of brain function.

Recent advances in electrode designs and implantation techniques have made ECoG more feasible for human applications [15]. The major benefit of this approach is that although itself being an invasive modality, this method does not lead to permanent neuron damage or perfusion issues. Moreover, since the modality is localized in nature, most of the noise is automatically canceled out. Nevertheless, as the modality cannot be coupled with any other modality to provide cross-cutting simulations, a complete coverage of all relevant brain events is not possible while conducting a study using this modality, thus limiting its adaptation.

The primary advantage of ECoG over other modalities, such as electroencephalography (EEG), lies in its superior resolution. Studies have indicated that ECoG encompasses a broader frequency range, enabling a more nuanced analysis of brain activity. In one study, voltage spectra derived from auto-regressive spectral analyses between 0 and 200 Hz were analyzed [16,17]. Each task was compared to the inactive period, and the R² values were calculated to determine the changes in mu, beta, and gamma rhythms associated to signals captured by electrodes strongly correlated with directional movements. It was demonstrated in this study that ECoG has a far broader frequency range than that of EEG. ECoG has been successfully applied to subjects suffering from instances of drug-refractory epilepsy [15]. Figure 2 shows ECoG study for finger activation.

Data Acquisition Approaches and Limitations with a Standalone Setup

ECoG is a method of acquiring electrical signals directly from the surface of the brain, providing high spatial and temporal resolutions that are crucial for BCI research. In a standalone setup, ECoG involves the placement of a grid of electrodes on the cortical surface, under the skull, to record neural activity associated with motor and cognitive functions. This setup allows for the real-time decoding of motor intentions and speech-related signals, making it particularly valuable for applications such as the control of prosthetic devices and speech rehabilitation in patients with communication impairments [18,19].

Despite its advantages, ECoG faces several limitations when used in a standalone configuration. The most significant challenge is the invasiveness of the procedure, which requires a craniotomy to place the electrodes on the brain’s surface [20]. This surgical requirement increases the risk of complications, such as infections and hemorrhaging, and limits the number of potential candidates for this technique. Furthermore, while ECoG provides a better spatial resolution and frequency range compared to non-invasive modalities like EEG, its inability to be combined with other modalities restricts the scope of neural data that can be captured. This limitation poses challenges in obtaining a comprehensive understanding of brain activity, especially when studying complex cognitive tasks that may require multimodal data integration. Additionally, electrode drift and long-term stability issues, similar to those encountered in other invasive recording methods, can affect the consistency of signal acquisition over time [21].

Recent advancements in electrode design and implantation techniques have mitigated some of these risks, making ECoG a more viable option for human applications [22]. However, the standalone nature of the setup means that the data processing and analysis capabilities are constrained by the hardware’s limitations, which may not always be sufficient for complex, real-time BCI applications [23].

2.3. Electroencephalography (EEG)

Electroencephalography (EEG) stands as a prominent non-invasive brain-imaging modality, being characterized by an array of electrodes adorning the scalp that diligently capture potential differences across distinct neuronal enclaves. Within the realm of brain–computer interfaces (BCIs), EEG assumes a pivotal role, with its operational architecture often tailored to specific frequency bands.

In the conventional BCI configuration, subjects are presented with stimuli at precise intervals. These stimuli elicit potentials within targeted cerebral regions, constituting the renowned event-related potentials (ERPs). The deliberate repetition of these ERPs enhances the signal-to-noise ratio, attenuating extraneous interference. Noise, discerned as spurious signals originating from artifacts such as cardiac activity, cranial structures, and respiratory processes, is systematically minimized. Recently, biometric authentication systems have been also explored with EEG modality banking on unique behavioral patterns to identify the user [24].

The versatile applications of EEG-based BCI systems traverse a myriad of domains, ranging from prosthetic innovations to immersive video game experiences. This owes much to the modality’s non-invasiveness, rendering it conducive to widespread use. With streamlined setup procedures and the absence of surgical prerequisites, EEG modalities emerge as the vanguard for forthcoming BCI explorations across multifarious scenarios. Also, EEG has been implemented in automated driving setups that check for driver drowsiness [25]. Within the realm of neuroimaging techniques, electroencephalography (EEG) stands as a beacon of distinct advantages, setting it apart from its counterparts. A cascade of benefits distinguishes EEG, rendering it a quintessential tool for deciphering the intricacies of brain function.

Foremost is the unparalleled temporal resolution that EEG confers. This modality operates on a millisecond timescale, unrivalled by other neuroimaging techniques such as fMRI. This temporal acuity empowers EEG to unravel the chronology and dynamics of neural processes with exquisite precision. It excels in capturing the intricate temporal orchestration of cognitive phenomena and the precise timing of sensory perceptions. EEG’s low cost and portability amplify its prowess. In the financial realm, EEG equipment remains economical, facilitating its widespread adoption across research settings. The added dimension of portability imbues EEG with remarkable flexibility, allowing seamless transitions between disparate locations. From hospital wards to research laboratories, and even field studies, EEG offers a versatile lens through which to examine brain function in diverse contexts. Furthermore, machine learning algorithms have found quantitative applications [26] for outcome predictions in traumatic brain injury [27,28], eating disorders [29], and autism spectrum disorder [30] cases.

Safety emerges as a concern in neuroimaging, and EEG’s non-invasiveness and radiation-free nature allay these apprehensions. Its innocuous profile renders it suitable for an extensive spectrum of subjects, including children and expectant mothers. This safety assurance complements EEG’s unobtrusiveness, fostering an environment conducive to exploring neural dynamics. A hallmark attribute of EEG lies in its capability to discern neural responses to specific stimuli or events. This attribute is pivotal in dissecting cognitive processes like attention, memory, and language. By mapping brain activity onto cognitive functions, EEG empowers researchers to unveil the intricate interplay between neural patterns and cognitive operations. EEG shines as a formidable neuroimaging tool, armed with an arsenal of advantages that demarcate its uniqueness. EEG’s temporal precision, portability, and responsiveness to stimulus-driven changes enable detailed investigations into brain function. Among various neuroimaging methods, EEG stands out by providing a unique perspective for analyzing neural activity.

Among the software options available, we have found python and R to be rich in libraries that aid in data filtering and analysis in the course of this review, such as the MNE-Python [31] and nin-Py [32] modules with a multitude of functionalities. Figure 3 shows calibration of EEG readings in R workspace. The topographic maps (Ψ) display the spatial distribution of electrical activity across the scalp in microvolts (µV) for different ERP components. The color scales, ranging from −3 µV to +9 µV, provide a standardized representation of voltage levels, with warmer colors (red) indicating higher positive voltages and cooler colors (blue) representing negative voltages. The temporal waveforms (τ) illustrate the evolution of these components over a time sequence from 0 to 1 second after stimulus onset. The x-axis represents time in seconds, while the y-axis shows amplitude in microvolts, with a range from −1 µV to +1 µV. These waveforms demonstrate the distinct temporal characteristics of each component, highlighting peaks and troughs associated with neural responses to stimuli. The pairing of Ψ and τ for each component allows for an integrated understanding of both the spatial and temporal dynamics of the ERP signals. Together, these representations capture the interaction between neural activity’s topography and time course, providing insights into brain processes underlying sensory and cognitive functions.

Data Acquisition Approaches and Limitations with a Standalone Setup

EEG-based brain–computer interfaces (BCIs) epitomize a revolutionary paradigm, channeling EEG signals to govern external devices such as computers or prosthetic counterparts. This dynamic symbiosis hinges upon the discernment of distinct brain activity patterns aligned with specific commands. While EEG-based BCIs yield manifold advantages, their evolution is also marked by intrinsic constraints warranting scrutiny. Non-invasiveness and high temporal resolution stand as hallmarks of EEG-based BCIs. These attributes empower seamless interaction with the brain’s cognitive canvas. However, a caveat arises in the form of spatial resolution. EEG’s measurement of electrical activity on the scalp distills activity remote from the neural core. The resultant signals traverse a labyrinth of influences, from skull thickness to scalp conductivity, subtly tinging measurement accuracy.

Figure 4 shows a raw EEG data profile as available through eegkit library of R. For the sake of consistency, we have retraced their steps with EEG signal data as they did with fNIRS data. We observed that a dedicated library such as eegUtils might also have been used to attain similar results. It must be noted that such dedicated libraries are also capable of providing additional topologies regarding the dataset versus its mapping in the brain. However, such niche results come at the cost of specializing the framework for a specific methodology and lead to loss of general capabilities in the BCI system.

Susceptibility to artifacts emerges as a thorny challenge in EEG-based BCIs. The susceptibility to contamination by diverse noise sources—be it motion artifacts, eye blinks, or muscular contractions—complicates the demarcation between genuine brain signals and confounding interference. This reality introduces an intricacy that demands judicious signal processing.

Effective command control in EEG-based BCIs hinges upon rigorous user training. Mastering the generation of specific neural patterns linked to designated commands, be it cursor movement or prosthetic limb control, demands substantial effort and time investment. This learning curve underscores the cognitive effort required to harmonize intent with action. A critical limitation lies in EEG-based BCIs’ reduced information content compared to counterparts like fMRI. While fMRI unveils brain region localization and activation specifics, EEG merely captures scalp-level electrical dynamics. This restricted information domain can hinder the decipherment of intricate cognitive processes or the precise identification of implicated brain regions.

An ICA Model has been considered for the purposes of this review, with source signals designated by columns S, the mixing matrix is denoted by columns M, and the noise signals are represented by E, whereas zero mean columns are denoted by X. Its aim is to usually compute the unmixing matrix W in a manner that tcrossprod (X, W) denoted by Column S are as independent from other columns as possible. The goal is to find the orthogonal rotation matrix R such that the source signal is able to estimate S (=Y% × %R) [33] and is as independent as possible [34]. Additionally, the Infomax approach is able to calculate the orthogonal rotation matrix R that nearly maximizes the joint entropy of a nonlinear function. Also, the orthogonal rotation matrix R is computed using the FastICA algorithm that, again, nearly maximizes the negentropy of the estimated source signals. Figure 5 displays brain maps after signal analysis and noise-filtering activities.

In summation, EEG-based BCIs marshal a transformative narrative by harmonizing brain and machine. Nonetheless, their trajectory is shadowed by limitations. The challenge of spatial precision, the battle against artifacts, the exigent training requirements, and the information ceiling: these facets underscore the need for a judicious balance between potential and constraints. Despite these limitations, EEG-based BCIs hold their ground as potent tools across diverse domains, propelling scientific inquiry in medicine, neuroscience, and human–computer interaction. The pursuit of innovation in this intersection remains unwavering as researchers strive to chart new vistas and conquer existing limitations in the landscape of EEG-based BCIs.

2.4. Magnetoencephalography (MEG)

MEG works by recording magnetic fields produced by potential differences in neurons using ultra-sensitive magnetic field sensors [35]. This modality has a better spatial resolution than other non-invasive modalities. The working mechanism involves the passage of electric current within a strong magnetic field that interferes with the intracellular current of the dendrites, thus mapping their structure and features. Figure 6 shows a schematic diagram of the workings of MEG.

However, high external magnetic noise in a general setting prevents the application of this modality in a real-life scenario. Nevertheless, active research is being conducted to ensure the viability of this modality as a high-resolution imaging system that might help in creating better commercial BCI products. Not much data analytics has been performed on this modality at the moment, and simple spatial filtering methods have been incorporated that take the properties from a geometric viewpoint of signal transmission in MEG into account and process artifacts specifically encountered in MEG-based BCIs. Such techniques have enabled researchers to achieve real-time synchronized feedback of stimuli against the actions performed [37]. Figure 7 shows a schematic diagram of a MRI-scanner labelled with its various components.

More recently, EEG and MEG [35] are also being analyzed together with fMRI to produce better temporal resolution in the results. A combination of the results from all three is used in a study of memory processing, wherein the details of the localization, activation, and time course of a specific area of activation could be determined accurately [38]. Moreover, to counter the slow speed of the readings, acceleration techniques are applied to shallow learning [39], deep learning [40,41,42,43,44], and ensemble learning [45,46,47,48,49,50] setups. A software implementation for data classification of fMRI has been attempted within the Grid Workflow environment. In this paper, we attempt to review a modality encompassing a Grid Workflow setup that can be easily adapted and scaled with any modality added with AI/ML capabilities.

Data Acquisition Approaches and Limitations with a Standalone Setup

Data acquisition in MEG involves capturing the weak magnetic fields generated by neuronal activity using highly sensitive sensors, typically superconducting quantum interference devices (SQUIDs). These sensors are arranged in an array around the subject’s head to ensure comprehensive spatial coverage of the brain’s magnetic activity. The process requires meticulous calibration and environmental control to minimize interference from external magnetic sources and to enhance the fidelity of the acquired data.

Given the extreme sensitivity of MEG sensors, data acquisition in a standalone setup often necessitates the use of magnetically shielded rooms (MSRs). These rooms are designed to attenuate external magnetic fields that could otherwise overwhelm the weak signals originating from neuronal activity. The effectiveness of shielding is paramount in ensuring the quality of the data, particularly when the setup is intended for use outside controlled laboratory environments.

Synchronizing the data acquisition process with external stimuli or other neuroimaging modalities (e.g., EEG or fMRI) is essential for capturing precise temporal correlations in brain activity. In standalone MEG setups, this often involves the use of high-precision clocks and synchronization protocols to ensure that the timing of data acquisition is accurately aligned with the experimental conditions.

The standalone operation of MEG systems requires a high degree of technical expertise to manage the calibration, data acquisition, and noise mitigation processes. This complexity can be a barrier to the widespread adoption of MEG outside of specialized research institutions. Additionally, the need for constant monitoring and adjustment during data acquisition increases the operational burden, potentially limiting the feasibility of using MEG in routine clinical or commercial settings.

2.5. Functional Magnetic Resonance Imaging (fMRI)

Functional magnetic resonance imaging (fMRI) stands as a pre-eminent neuroimaging methodology, unravelling the intricate tapestry of brain activity through the lens of hemodynamic response. This technique leverages the relationship between neural activation and metabolic requirements, facilitating an understanding of the neural mechanisms underlying cognitive and motor functions. At its core, functional magnetic resonance imaging (fMRI) serves as a sophisticated tool for capturing the dynamic changes in blood flow within the brain, which are indicative of underlying neural activity. This phenomenon is primarily driven by the increased metabolic demands of activated neurons. When specific regions of the brain engage in cognitive or sensory processing, there is a corresponding rise in the consumption of oxygen and nutrients. This heightened metabolic requirement stimulates a localized increase in cerebral blood flow, a process known as neurovascular coupling. This metabolic upsurge translates to an augmentation in blood flow, an effect termed the hemodynamic response.

In practice, fMRI unfolds as a collaborative endeavor between the subject and the scanner. The subject engages in tasks while the scanner, fortified by a potent magnetic field and radio waves, tracks blood flow oscillations. These oscillations metamorphose into vivid images of neural engagement, portraying the choreography of brain activity. This visual revelation lays bare the precise regions orchestrating a given task and unfurls the intricate interplay between these enclaves. A prominent asset of fMRI lies in its exquisite spatial resolution, a hallmark trait that unfurls the neural canvas in high definition. With spatial precision bordering on a few millimeters, fMRI excels in the cartography of brain regions. This capability assumes particular importance in decoding the neural circuits engaged in cognitive and motor tasks. Insights gleaned from fMRI have been pivotal in deciphering neural circuitry in various contexts, be it cognition, motion, or the perturbations induced by afflictions.

The non-invasive nature of fMRI makes it a widely embraced and secure tool for probing the human brain. Unlike invasive techniques, fMRI circumvents surgical interventions, cementing its status as a low-risk procedure. This attribute equips researchers to navigate the neural landscapes of healthy individuals and patients grappling with neurological or psychiatric maladies. In concert with its spatial precision and safety, fMRI unfolds its versatility by embracing real-time exploration of brain function across a diverse spectrum of cognitive and motor activities. This expansive reach empowers fMRI to stand as a versatile sentinel, casting light upon the neural substrates of behavior and cognition.

Functional magnetic resonance imaging (fMRI), in particular, stands out as a pivotal tool in this framework. It employs powerful magnetic fields to capture the intricate patterns of neural activity, effectively mapping the dynamic interplay of brain functions over time. The exceptional spatial resolution and non-invasive nature of fMRI render it invaluable in both clinical and research settings. This modality allows for the visualization of brain activity with remarkable clarity, thereby illuminating the neural underpinnings of various cognitive and emotional processes. Through fMRI, researchers can delve into the complexities of brain health and disease, unveiling the subtle changes that characterize neural disorders and informing the development of targeted therapeutic interventions. Figure 8 shows conventional fMRI analysis examining the BOLD response to target stimuli, illustrating how the signal measured in a specific voxel varies over time.

Data Acquisition Approaches and Limitations with a Standalone Setup

Functional magnetic resonance imaging (fMRI) stands as a pioneering tool in forging brain–computer interfaces (BCIs) that bridge the chasm between mental states and external device control. While fMRI-based BCIs unveil a constellation of strengths, they also tread a path punctuated by nuanced limitations. The fusion of fMRI with BCIs heralds a realm of promise, underscored by remarkable advantages. Paramount among these is fMRI’s exceptional spatial resolution, unveiling neural orchestrations with exquisite detail [51]. Additionally, fMRI’s unique prowess in peering into the depths of intricate brain structures enriches our understanding beyond the purview of alternative techniques.

However, this is accompanied by constraints warranting judicious contemplation. The temporal resolution of fMRI emerges as a limitation, capturing neural dynamics at a deliberate pace spanning seconds. Consequently, rapid mental state transitions, such as those underpinning motor control or language processing, prove challenging for fMRI-based BCIs. Further, fMRI’s susceptibility to motion artifacts introduces a layer of complexity that perturbs certain BCI applications. The landscape of fMRI-based BCIs is further shaped by the complexity and costs they entail. The procurement and upkeep of fMRI scanners entail substantial financial investments, alongside the need for specialized personnel to navigate their operation. This amalgam renders the translation of fMRI-based BCIs into everyday environments, like homes or workplaces, an intricate endeavor.

Ethical considerations add a profound layer of discourse. The ability of fMRI to divulge an individual’s mental state raises ethical dilemmas about privacy and consent. The potential misuse of this sensitive information poses concerns of unwarranted inference into personal thoughts or intentions. This interplay of technological prowess and ethical stewardship underscores the need for a conscientious balance [52]. Amid these contours, fMRI-based BCIs showcase encouraging strides in areas ranging from motor control to communication and cognitive enhancement. The trajectory of relentless research and development augurs a future where fMRI-based BCIs will seamlessly intertwine with clinical and research realms, enriching our understanding and shaping human–computer interaction.

The alliance between fMRI and BCIs offers an avenue of exploration, charting the fusion of neural states and external control. However, the expedition is not devoid of challenges. The careful calibration of potentials and limitations navigates a course toward a future where fMRI-based BCIs coalesce with ethical diligence, technical ingenuity, and transformative application. As this journey unfolds, the horizon of fMRI-based BCIs shines with the promise of shaping a dynamic nexus between human cognition and technological advancement.

There is a push towards the development of non-invasive techniques for functional imaging of the brain. fMRI displays contrast differences in the Blood Oxygen Level-Dependent (BOLD) signal, wherein the differential contrast is observed due to variable blood flow in the grey and white matter, where the latter receives greater blood flow. Figure 9 represents fMRI readings from a 3D image acquired for 152 time points.

2.6. Functional Near-Infrared Spectroscopy (fNIRS)

Functional near-infrared spectroscopy (fNIRS) emerges as a pivotal non-invasive neuroimaging modality that provides insights by measuring near-infrared light interactions with cerebral blood flow. Its temporal resolution allows researchers to capture real-time brain activity. This method provides a practical approach to observing cognitive functions as they occur. Rooted in the tenet that neural activity is entwined with hemodynamic shifts, fNIRS deciphers the neural script by monitoring the absorption of light within the brain. fNIRS is another non-invasive modality that is performed by imaging hemodynamic signals of brain tissues using the near-infrared spectrum. It works by measuring hemoglobin concentration changes in the blood stream [53,54]. Distinct virtues mark fNIRS as a noteworthy contender in the neuroimaging arena. Its non-invasive nature, devoid of ionizing radiation, resonates as a harbinger of safety, making it universally palatable. The harmony between neural activity and hemodynamics finds resonance in fNIRS, forging a pathway to trace the ebb and flow of blood. This orchestration is unveiled by detecting fluctuations in the concentrations of oxygenated and deoxygenated hemoglobin, unraveling the neural narrative. Moreover, fNIRS is able to provide a better temporal resolution [55,56]. Figure 10 shows various components of an fNIRS device. The fNIRS sensor has been equipped with 4 light source centers and 10 detectors at the top along with 16 locations for optode measurement that are registered on the sensor.

The capabilities of fNIRS are marked by distinct advantages. Notably, its temporal resolution allows researchers to capture real-time brain activity. This method provides a practical approach to observing cognitive functions as they occur. This modality is used in locomotory and ambulatory studies including yoga, meditation [58], robotics [59], and emotional [60] therapies. fNIRS is a virtuoso in unveiling the intricacies of deep brain structures that often evade other neuroimaging methodologies. The prefrontal cortex and superior temporal gyrus, enclaves of cognitive richness, become accessible realms under fNIRS’s gaze. Additionally, its robustness against movement artifacts renders it a preferred choice in studying infants and young children, circumventing the constraints that encumber EEG or fMRI. fNIRS stands out against fMRI in its portability and ability to filter out noise. Although it provides a lower spatial resolution, fNIRS offers higher temporal resolution and allows measurements of hemodynamic concentration changes with changing the exertion of muscles [61].

In concert with other neuroimaging techniques, functional near-infrared spectroscopy (fNIRS) emerges as a pivotal modality, adept at unraveling neural narratives in real time with remarkable temporal fidelity. Its non-invasive and portable nature positions fNIRS as an exceptionally versatile tool, finding applications across various domains, including clinical investigations, neurofeedback training, and the burgeoning field of brain–computer interfaces (BCIs).

The utility of fNIRS lies in its ability to monitor hemodynamic responses by measuring changes in the concentration of oxygenated and deoxygenated hemoglobin within cortical tissues. This capability allows for the exploration of cerebral activity, providing insights into the intricate dynamics of brain function during cognitive processes and emotional responses. Amidst the array of neuroimaging methodologies, the unique attributes of fNIRS resonate with clarity, tracing the cerebral rhythms that contribute to our cognitive tapestry. Each neuroimaging modality, including fNIRS, operates through a series of systematic stages encompassing data acquisition, preprocessing, and feature selection [62]. These stages are critical for filtering and classifying input data, effectively mitigating noise while amplifying relevant signals. The preprocessing phase involves techniques such as motion artifact correction and physiological noise reduction, ensuring the integrity of the data collected. Subsequently, feature selection identifies pertinent characteristics from the acquired signals, enhancing the accuracy and reliability of the analyses. By integrating fNIRS with complementary imaging techniques, researchers can achieve a more comprehensive understanding of neural processes, thereby enriching the narrative of brain activity and its implications for both research and clinical applications.

Figure 11 shows a demonstration of R-fNIRS brain network reliability analysis. The fNIRS workflow involves several key steps. First, researchers formulate their research question and design the study. Participants are then recruited and consent to it. In the pre-experiment phase, fNIRS probes are placed on the participant’s scalp and calibrated if needed. During the experiment, data are collected while participants perform tasks or experience stimuli. Subsequently, the collected fNIRS data are preprocessed to remove noise and transformed into hemoglobin concentration changes. Statistical analyses are applied to compare these changes between conditions. Interpreting the results involves relating hemodynamic patterns to the research question and the existing literature. The implications and limitations of findings are discussed before drawing conclusions and considering future research directions.

Data Acquisition Approaches and Limitations with a Standalone Setup

Functional near-infrared spectroscopy (fNIRS) emerges as a burgeoning modality in the realm of brain–computer interfaces (BCIs), leveraging its hallmark attributes of exceptional temporal resolution and portability. In the harmonious interplay between fNIRS and BCIs, insights into neural dynamics are harnessed, propelling interactions between cognition and external devices.

fNIRS-based BCIs harness the dynamic landscape of blood oxygenation changes within the brain to decipher distinct mental states. This intimate connection between hemodynamics and cognition serves as the foundation for translating these changes into control signals, enabling users to seamlessly navigate computer interfaces and devices. The allure of fNIRS-based BCIs rests in their non-invasive nature, imbuing comfort and accessibility. Their portability further amplifies their utility, flexibly extending from clinical settings to diverse environments.

We have subsequently reviewed data processing by taking publicly loaded datasets against these action points and pushed them over R engine to visualize the datasets, as shown in Figure 12. One of the limitations of this library is that this package can only read raw csv files generated by the Hitachi ETG-4000 [63]. However, the package is under active development, and further development of additional support for different file types that looks promising. Both R and python provide libraries with similar syntax that give almost equivalent output. However, python versions are more mature and are able to provide finer results as compared to their R counterpart. The performances of both software packages are equivalent to each other without any notable difference. However, it should also be noted that for graphical user interfaces, R programs written in Shiny are much faster than the python graphical interface programs written in wx-widgets or tcl-Tk.

The data are subsequently segregated and normalized in the R workbench. We have used SparklyR functions and the fnirsr library designed by Eryk Walczak [63] to review the BCI analytics workflow. However, alternate functions can also be used. The data are further subjected to cleaning and preprocessing to obtain noise-filtered clean channels, as observed in Figure 13. The fNIRS signal is likely to show a linear trend that can be removed. The linear trend can be removed from all channels (recommended) or from a single channel. Additionally, the library also supports HOMER2 datasets that provide additional information that can help a researcher visualize the data in faceted time-series plots. Using the functions, a graph can be denoised and merged to create a comparative study, as shown in Figure 14. Profiling methods with publicly available EEG data are available, just like it was performed for an fNIRS dataset [64,65]. HOMER2 [66] and OptoNet II [67] provide the MATLAB scripts used for analyzing fNIRS data. The libraries are able to provide estimates and maps of brain activation areas. The application has been in a constantly evolving state since the early 1990s. It started off as the Photon Migration Imaging toolbox, which was subsequently reshaped into HOMER1. The application has undergone considerable changes to be rebranded as HOMER2. The application has a GUI interface similar to its older version but with easier and better support for group analyses and re-configuration of the processing stream. Also, it allows users to integrate their custom algorithms into the processing stream.

The appeal of functional near-infrared spectroscopy (fNIRS)-based brain–computer interfaces (BCIs) stems from their portability, non-invasiveness, and capability to provide real-time insights into neural dynamics. However, the path forward is marked by the imperative to enhance specificity, accuracy, and robustness against noise and artifacts. Future research must navigate a complex array of technological and methodological challenges. Advancing research in fNIRS-based BCIs will require addressing significant technological and methodological barriers to further illuminate the complexities of cognitive processes.

In the ongoing evolution of fNIRS-based BCIs, each limitation presents a compelling opportunity for innovation. As researchers refine the precision of spatial localization and classification accuracy, and as methodologies to minimize noise and artifacts advance, the synergistic relationship between fNIRS and BCIs is poised to flourish. This progression will cultivate a landscape where cognitive intention seamlessly translates into action, characterized by unprecedented clarity and fidelity. Furthermore, interdisciplinary collaboration among neuroscientists, engineers, and clinicians will be crucial in addressing these challenges. By fostering an integrative approach that combines insights from various fields, the development of fNIRS-based BCIs can accelerate, paving the way for novel applications in rehabilitation, assistive technologies, and enhanced human–computer interaction. Ultimately, as fNIRS technology continues to evolve, it holds the promise of not only advancing our understanding of neural dynamics but also enhancing the quality of life for individuals with neurological impairments.

2.7. Multi-/Single-Unit Activity (MUA/SUA)

Multi-unit activity (MUA) and single-unit activity (SUA) are critical components in the study of neural signals, providing insights into the collective and individual behaviors of neurons, respectively. MUA refers to the electrical activity recorded from a group of neurons in proximity to an electrode, capturing the summed action potentials from multiple neurons [68]. This modality is particularly useful for understanding the overall neural activity within a specific brain region, as it provides a broad view of neuronal ensemble dynamics. On the other hand, SUA focuses on the activities of individual neurons, offering a more granular perspective by isolating and recording the action potentials from a single neuron. This is typically achieved through the use of high-impedance microelectrodes that can discern the firing patterns of individual neurons from the surrounding neural noise. The combination of MUA and SUA in neural recordings, often facilitated by microelectrode arrays (MEAs), allows researchers to study both the macro-level population dynamics and micro-level neuronal specificity, contributing to a more comprehensive understanding of brain function [68]. These modalities are crucial in various applications, including brain–computer interfaces (BCIs) and the study of neural circuits in both normal and pathological states.

3. Integrated Setup for Advanced BCI

As the field of advanced BCIs continues to evolve, we can expect to see exciting applications that not only enhance our understanding of the brain but also improve the lives of individuals with neurological conditions, create more immersive entertainment experiences, and revolutionize human–computer interaction. The future of BCIs is filled with promise, and ongoing research and development will play a critical role in unlocking their full potential. Creating an integrated setup for advanced BCIs that combines various standalone BCI modalities involves careful planning, specialized equipment, and robust synchronization methods. Neurovascular coupling is the phenomenon of a rise in cortical blood flow. It is a multi-step activity that encompasses all actions, from the stimulation of neurons to the release of transmitters, that finally lead to vasoconstriction/dilation. Although associated with the same activity, EEG and fNIRS cover separate events in this cascade. The collaboration of these two methods offers a unique chance to examine cortical activity in an elaborate manner. EEG and fNIRS have very different but complementary temporal and spatial resolutions. On the one hand, cortical responses having high temporal resolution are detected by evoked potentials to a given stimulus; on the other hand, fNIRS depends on the localization of changes in the metabolism of hemoglobin with oxygen post-activation. The combination of these modalities allows for a comprehensive examination of neurovascular coupling. For instance, EEG-fNIRS and MEG-fMRI setups enable researchers to simultaneously capture the fast electrical activity of neurons and the slower hemodynamic changes, offering a detailed view of the timing and spatial distribution of brain activity. These multimodal approaches are particularly powerful in understanding how neuronal and vascular components interact in both normal and pathological conditions. Placement sensors can be achieved in different manners. The first method is called adjacent positioning, whereas the second method is called co-located measures. This method is limited to ring electrodes and requires transparent gel, which are usually non-conductive. As fNIRS and EEG measurements are recorded independently, it is important for simultaneous trigger synchronization with both data streams [69]. One of the key advantages of combining these modalities is the complementary nature of their spatial and temporal resolutions. EEG and MEG provide millisecond-scale temporal resolution, capturing the immediate neuronal responses to stimuli. In contrast, fMRI and fNIRS offer millimeter-scale spatial resolution, mapping the ensuing hemodynamic changes across the cortex. Together, they provide a multi-faceted view of brain activity, linking neuronal events to vascular responses. The combination of MEA, ECoG, EEG, MEG, fMRI, and fNIRS provides a powerful toolkit for investigating neurovascular coupling. Each modality offers unique strengths, and their integration allows for a more comprehensive understanding of the complex interactions between neuronal activity and blood flow in the brain.

3.1. EEG/MEG-fMRI/fNIRS Integration

Combining electroencephalography (EEG) and magnetoencephalography (MEG) with functional magnetic resonance imaging (fMRI) or functional near-infrared spectroscopy (fNIRS) provides a comprehensive view of brain activity by leveraging the strengths of each modality. EEG and MEG offer excellent temporal resolution, capturing neural dynamics in real time, while fMRI and fNIRS provide superior spatial resolution, allowing for the detailed mapping of brain regions involved in various cognitive tasks.

The integration of these modalities is particularly valuable in studying complex brain functions such as memory processing, language, and motor control. For instance, EEG/MEG can detect fast neural oscillations that occur during cognitive tasks, while fMRI/fNIRS can localize the brain regions responsible for these activities. This multimodal approach enables researchers to determine not only where brain activity occurs but also when it happens, offering a more complete picture of brain function. An exact synchronous action is attained with the NIRx Parallel Port Replicator, which is used to stimulate the incoming signal via USB ports and splits a single DB-25 (parallel port) input to four or more outputs. A general setup for a representative fMRI-EEG hybrid setup is shown in Figure 15.

This figure presents a detailed schematic of the bimodal neurofeedback platform that integrates EEG and fMRI modalities for enhanced brain activity monitoring and feedback. It must be noted that MEG functions similar to EEG, so the two modalities can be readily substituted with each other. The diagram illustrates the flow of inputs from both fMRI and EEG systems, which are processed to ensure high-quality data integrity.

Initially, the input signals undergo artifact correction to eliminate any noise that may interfere with the subsequent analysis. Following this, various filtering techniques are applied to refine the signals further, ensuring that only relevant data are retained for processing. The activation maps generated from these modalities provide critical insights into neural activity, while the neurofeedback (NF) features extracted from the data facilitate personalized feedback based on the user’s brain state.

Ultimately, the processed output is displayed on a visualization screen within the MRI room, enabling the real-time monitoring of and interaction with the neurofeedback system. This integration not only enhances the understanding of brain dynamics but also supports applications in cognitive and clinical settings.

Similarly, EEG records brain activity via external electrodes placed on the scalp, whereas fNIRS measures changes in hemoglobin concentrations using light sensors, albeit placed at similar locations on the scalp. EEG signals are filtered to remove electrical noise mostly using high- and low-pass filters. On the other hand, fNIRS filters correct for motion artifacts and physiological noise, such as cardiac and respiratory rhythms. During the feature-extraction step, EEG extracts features like spectral power, event-related potentials (ERPs), or connectivity measures, whereas fNIRS extracts features such as changes in oxygenated and deoxygenated hemoglobin levels. All these show that combining EEG and fNIRS does not lead to any conflicts; however, it can help alleviate the shortcomings of both. EEG and fNIRS can be combined to capture complementary information about neural and hemodynamic activities. Fusion techniques can also be applied that include concatenation, weighted averaging, or using them as inputs for a machine learning model. A combined schematic is presented in Figure 16. This figure illustrates the comprehensive workflow involved in the EEG-fNIRS hybrid setup for brain activity analysis. The diagram delineates the stages of signal acquisition, where both EEG and fNIRS signals are obtained concurrently to capture complementary data regarding neural activity and hemodynamic responses. Following acquisition, the preprocessing phase addresses noise reduction and artifact removal, ensuring the integrity of the collected signals. Subsequently, the feature-extraction stage identifies relevant characteristics from both EEG and fNIRS data, highlighting distinct features inherent to each modality. The feature fusion step integrates these extracted features into a unified dataset, enhancing the richness of the information available for analysis. Finally, the classification phase employs advanced algorithms to interpret the fused features, facilitating improved accuracy in understanding brain dynamics and function.

3.1.1. Data Processing in a Hybrid Setup

eegkit [71] is an R library that is powerful and provides precise results similar to its Python counterpart. It is a toolkit for electroencephalography data that uses a fast-discrete Fourier transform (eegfft) to calculate the power spectral density of EEG data and generates a plot of the power estimate using the plot (single-channel) or image bar (multi-channel) function. The function employs stratigraphic series for bandpass filtering. Generally, the first column is designated as the location (e.g., depth), whereas the second column ought to be a data value. Additionally, there is a plethora of other functions that perform multiple functionalities. Padfac function is used to Pad a specific column with zeros to (padfac × npts) points, where npts is the original number of data points. Flow gives the lowest frequency to bandpass, and fhigh provides the highest frequency to bandpass. win designates the Window type for the bandpass filter: 0 translates to rectangular, 1 to Gaussian, and 2 to a Cosine-tapered window, otherwise known as the Tukey window. Similarly, alpha, which translates as the inverse of the standard deviation and acts as a measure of the width of the Dirichlet kernel, gives the Gaussian window parameter [72]. Also, p provides the Cosine-tapered (Tukey) window parameter, where p is the percent of the data series-tapered parameter. The demean function is used to remove the mean from data series, whereas the detrend function removes the linear trend from the data series. Similarly, the addmean function is used to add the mean value to the bandpass result, and the output function is used to show the output for either the filtered series or bandpass filter window. The smallest frequency for plotting is determined using the xmin function, whereas xmax function gives the opposite, which is the largest frequency for plotting. A summary of plots can be visualized using the genplot function. Figure 17 displays the bandpass-filtered and normalized EEG data in R. The EEG signals were filtered using a fourth-order Butterworth bandpass filter with cutoff frequencies set between 0.5 Hz and 30 Hz to remove noise and retain relevant brain activity frequencies. The filtered data were subsequently normalized to a z-score, providing a mean of 0 and a standard deviation of 1. Different colors represent signals from various EEG channels, showing the dynamic changes in brain activity across time.

Classification models can be trained to use the integrated EEG-fNIRS features using labeled data from cognitive tasks to classify brain states or conditions. Similarly, the characterization of brain activity can be carried out by obtaining spatial information from EEG about localized neural activity. Then, fNIRS can be used to capture changes in oxygenation, indicating an increase or decrease in brain activity. These combined data can provide a more comprehensive view of cognitive processes.

3.1.2. Challenges with a Hybrid Setup

While fNIRS-based BCIs unveil a promising horizon, they are not devoid of limitations. One pivotal constraint lies in the modest spatial resolution of fNIRS, rendering the precise localization of neural activity a challenge in contrast to more robust techniques like fMRI. This intricacy poses a hurdle in the endeavor to attain specificity within BCIs.

Classification accuracy emerges as another arena of contention. While fNIRS-based BCIs have demonstrated potential in detecting mental states like concentration and relaxation, achieving a high accuracy for intricate tasks such as speech recognition or motor control remains elusive. The interplay of individual neural variations constrained spatial and temporal resolutions, and task complexity collectively contribute to this challenge. Noise and artifacts cast a further veil of complexity [73]. Motion artifacts and physiological noise introduce potential distortions that can erode the accuracy and reliability of fNIRS signals. Although signal processing and machine learning techniques offer mitigation strategies, the intrinsic vulnerability persists, necessitating ongoing vigilance.

3.2. fMRI-fNIRS Integration

The integration of fMRI and fNIRS data culminates in a detailed comprehension of brain function, harmonizing profound insights into neural dynamics furnished by fMRI with hemodynamic revelations from fNIRS. This dual-modality approach supports advancements in cognitive research, enhances clinical diagnostics, and improves our grasp of the dynamic interactions underlying neural activity and behavior. A synergy in signal acquisition can be achieved, as fMRI captures neural activity with the aid of powerful magnetic fields, whereas fNIRS quantifies hemoglobin concentrations by leveraging near-infrared light sensors. The alignment of fMRI and fNIRS data in spatial dimensions ensures their correspondence within specific brain regions. Co-registering these data enhances the fidelity of interpreting combined observations.

The complementary nature of these modalities is instrumental: fMRI utilizes magnetic resonance imaging to capture brain activity with detailed spatial resolution, mapping neural responses across defined anatomical regions. In contrast, fNIRS quantifies hemodynamic responses—specifically changes in hemoglobin concentration—through near-infrared spectroscopy, which offers real-time measurement of cortical activity. The spatial alignment, or co-registration, of fMRI and fNIRS data within specific brain regions improves the precision of neural activity interpretation by merging both modalities’ insights into a cohesive data framework.

This integration process also demands careful consideration of potential challenges, such as differing acquisition rates and resolution capabilities, which must be meticulously calibrated to avoid data misalignment. Furthermore, optimizing spatial and temporal alignment between these modalities may require advanced signal-processing techniques to minimize any temporal lag inherent in fMRI’s BOLD responses, alongside addressing depth limitations in fNIRS. Overall, these integration efforts foster a robust methodology for achieving a well-rounded, multi-faceted view of brain dynamics through combined neuroimaging.

3.2.1. Data Processing in a Hybrid Setup

Achieving temporal synchrony between fMRI and fNIRS data is crucial to accurately timestamp events. The amalgamation of fMRI attributes (e.g., BOLD activation maps) and fNIRS traits (e.g., oxygenation dynamics) generates a more nuanced perspective during the feature-extraction phase [74]. Employing advanced analytical techniques uncovers correlations and interplays between fMRI and fNIRS signals. Interpreting the unified dataset yields insights into the interplay between neural dynamics and hemodynamic alterations. Integrative fMRI-fNIRS configurations enable real-time neurofeedback, brain–computer interfaces (BCIs), and the examination of cerebral states during tasks. This also helps to address disparities in the spatial and temporal resolutions inherent in fMRI and fNIRS methodologies. Figure 18 displays a workflow of the hybrid fMRI-fNIRS setup.

3.2.2. Challenges with a Hybrid Setup

While the integration of fMRI and fNIRS offers significant advantages in capturing comprehensive brain activity [76], several challenges must be addressed to optimize the efficacy of this hybrid approach. One of the primary challenges is the disparity in spatial resolution between fMRI and fNIRS. fMRI provides high spatial resolution through the detection of BOLD signals, which can localize brain activity to specific cortical regions with millimeter precision. In contrast, fNIRS, while offering valuable hemodynamic information, generally has lower spatial resolution due to its reliance on near-infrared light passing through the scalp and skull [77]. This discrepancy can complicate the accurate co-registration of data, potentially leading to challenges in interpreting the spatial localization of brain activity.

The temporal resolution of fMRI and fNIRS also presents a challenge. fMRI measures BOLD signal changes with a temporal resolution in the order of seconds, which may not capture rapid neural fluctuations effectively. fNIRS, although it can track changes in hemodynamic responses with a finer temporal resolution, still does not match the millisecond precision of EEG or MEG. The integration of these differing temporal scales requires sophisticated synchronization techniques to ensure the accurate alignment of data across modalities.

Both fMRI and fNIRS are susceptible to various types of noise and artifacts. fMRI is particularly prone to motion artifacts, physiological noise (e.g., heartbeats and respiration), and scanner-related artifacts. fNIRS faces challenges related to motion artifacts, skin contamination, and physiological noise, such as variations in heart rate and respiration [78]. The presence of these artifacts can degrade the quality of the data and complicate the fusion process. Effective preprocessing and noise reduction techniques are essential to mitigate these issues.

The fusion of fMRI and fNIRS data involves complex data integration processes. These processes include the co-registration of spatial data, the alignment of temporal signals, and the application of analytical techniques to extract meaningful correlations. The complexity of these tasks demands advanced computational tools and methodologies to ensure the accurate and reliable integration of multimodal data.

3.3. ECoG-MEG/EEG Integration

The integration of electrocorticography (ECoG), a direct cortical recording method, with non-invasive modalities such as magnetoencephalography (MEG) and electroencephalography (EEG) enhances the spatial and temporal precision of brain activity monitoring, yielding a more detailed portrayal of neural dynamics that provides high spatial resolution and an exceptional signal-to-noise ratio due to the electrodes’ direct contact with cortical tissue, which enables the precise localization of brain regions engaged in specific functions. This is effective in clinical applications, such as pre-surgical planning for epilepsy patients, where precise cortical mapping is essential. In contrast, MEG and EEG are non-invasive techniques capable of capturing the temporal dynamics of neural activity with millisecond precision, although they have comparatively lower spatial resolution. Integrating ECoG with MEG/EEG allows researchers to benefit from ECoG’s spatial accuracy and signal fidelity while leveraging the broader cortical coverage and non-invasive nature of MEG/EEG. This multimodal approach facilitates a more comprehensive analysis of neural processes by enabling the precise localization and temporal tracking of brain events across multiple regions.

ECoG, an invasive method that records electrical activity directly from the cortical surface, can be integrated with non-invasive modalities such as MEG and EEG to enhance the spatial and temporal resolutions of brain activity monitoring. This multimodal approach leverages the strengths of each technique, providing a comprehensive view of neural dynamics [79]. This integration also requires careful alignment of data from each modality, given the differences in spatial sampling and temporal resolution. Advanced data-processing techniques, such as source-localization algorithms and coherence analysis, are often employed to synchronize and interpret combined signals accurately. By unifying these complementary modalities, researchers can achieve a nuanced and multidimensional understanding of brain function, improving both the specificity and scope of neural investigations.

ECoG offers a high spatial resolution and excellent signal-to-noise ratio due to its proximity to the neural tissue, making it particularly effective for localizing the brain regions involved in specific functions [80]. However, its invasive nature limits its application to clinical settings, particularly in patients undergoing epilepsy surgery. In contrast, MEG and EEG are non-invasive techniques with lower spatial resolution but provide critical information on the temporal dynamics of neural activity. By integrating ECoG with MEG/EEG, researchers can combine the high spatial precision of ECoG with the broader coverage and non-invasiveness of MEG/EEG, leading to the more accurate localization and timing of brain events. Figure 19 shows the ECoG-EEG/MEG hybrid setup. Co-registering the data enhances the fidelity of interpreting combined observations.

The top row provides definitions for Brainstorm inputs within a dataset, which may include individual structural MRI (sMRI) acquisitions, such as T1-weighted (T1w) and T2-weighted (T2w) images, or none of these acquisitions, alongside the MNI-registered sMRI template that best represents the dataset, and either EEG or MEG data acquisitions. The second row presents the structural processing pipeline, consisting of standard and legacy modules compatible with the Human Connectome Project (HCP). The third row includes the source model, head model, and lead field modules, each a component of the forward modeling pipeline. Finally, the fourth row contains the modules for prior information integration and the inverse solution within the inverse modeling pipeline. ECoG provides a high spatial resolution and signal quality, particularly for cortical areas, which can complement the broader coverage of MEG/EEG. Combining invasive ECoG with non-invasive MEG/EEG can result in richer datasets, improving BCI system robustness [82].

3.3.1. Data Processing in a Hybrid Setup

In a synchronous setup, ECoG, MEG, and EEG data are acquired simultaneously, allowing for the direct correlation of signals across different modalities [83]. This approach is particularly useful in clinical settings, where the real-time monitoring of brain activity is required. The challenge lies in the synchronization of the data streams from the different modalities [84], which often have different sampling rates and data formats. Advanced signal-processing techniques, such as time–frequency analysis and coherence analysis, are typically employed to align and integrate the data.

In some cases, ECoG and MEG/EEG data are acquired asynchronously, either due to technical constraints or the need to avoid excessive patient burden. While this approach may reduce the complexity of the experimental setup, it requires careful post-processing to align the data temporally. This often involves using external markers or events, such as auditory or visual stimuli, to synchronize the data retrospectively.

The combination of ECoG’s high spatial resolution with MEG/EEG’s excellent temporal resolution provides a detailed view of neural activity. This is particularly valuable in studies of brain function where both the precise location and timing of neural events are critical, such as in the investigation of seizure dynamics in epilepsy or the study of sensorimotor processes [83,85].

ECoG provides localized information from specific brain regions, while MEG/EEG covers a broader area of the cortex. The multimodal approach allows for the investigation of interactions between local and distributed brain networks, offering a more holistic understanding of brain function.

ECoG’s proximity to the cortical surface results in a higher signal-to-noise ratio compared to scalp-recorded EEG. Integrating ECoG with MEG/EEG can help validate findings from non-invasive recordings, reducing the likelihood of artifacts and improving the reliability of the results.

3.3.2. Challenges with a Hybrid Setup

The primary limitation of this multimodal approach is the invasiveness of ECoG, which requires the surgical implantation of electrodes. This restricts its use to clinical populations, particularly patients undergoing neurosurgical procedures [86]. The risks associated with surgery, such as an infection and hemorrhage, also limit the widespread application of ECoG.

Combining data from multiple modalities with different sampling rates, spatial resolutions, and noise characteristics presents significant technical challenges. Advanced computational tools are required to align, synchronize, and integrate the data, which can be computationally intensive and time consuming.

ECoG requires surgical implantation, limiting its use to clinical settings or specific patient groups. Moreover, integrating invasive and non-invasive data involves complex signal processing and interpretation challenges.

The multimodal setup is resource-intensive, requiring specialized equipment, software, and expertise in both invasive and non-invasive neuroimaging techniques. This can limit the accessibility of this approach to well-funded research institutions and clinical centers [87].

While MEG and EEG can be used for the long-term monitoring of brain activity, the invasive nature of ECoG typically restricts its use to shorter periods. This limits the ability to study long-term neural dynamics, such as those involved in learning and memory, using the multimodal approach [88].

3.4. MUA/SUA-MEA/ECoG Integration

The integration of MUA and SUA with MEA and ECoG provides a powerful framework for studying neural activity with high spatial and temporal resolutions. This hybrid approach allows for a comprehensive examination of neuronal dynamics, facilitating insights into both single-neuron and population-level activities. The subsequent subsections detail the data–processing techniques and challenges associated with this integration [89]. This multimodal approach expands the analytical capabilities in neuroscience, providing insights into distinct yet inter-related neural phenomena.

Integrating MUA/SUA with MEA/ECoG involves capturing neural signals from different sources with varying spatial resolutions. MUA and SUA are typically recorded using high-density electrodes or fine-tipped microelectrodes. MUA and SUA offer fine-grained information on neuronal firing patterns, which are typically recorded through high-density or microelectrode configurations that target single neurons (SUA) or small neuronal clusters (MUA). These recordings provide precise temporal data, which are essential for understanding rapid neuronal events. In contrast, MEAs capture activity across a broader spatial area, consisting of multiple electrode sites that facilitate the monitoring of population-level dynamics. ECoG, positioned directly on the cortical surface, offers high-resolution spatial and temporal data. Achieving temporal synchronization between these modalities is crucial for accurate data fusion. Techniques such as timestamp alignment and event-based triggering are employed to ensure that data from MUA/SUA, MEA, and ECoG are temporally coherent. Figure 20 displays the various points on the skull where a hybrid setup can be arranged.

This multimodal setup enables researchers to observe neural dynamics with greater spatial and temporal precision, capturing both single-neuron events and broader cortical activity patterns. This integration holds significant promise for applications requiring fine temporal resolution and expansive spatial coverage, such as studies on network connectivity and real-time brain–computer interfaces. Figure 20 illustrates possible electrode placements on the cortical surface, demonstrating the arrangement of MUA/SUA, MEA, and ECoG electrodes in a hybrid setup. Various BCI sensor form factors are designed for placement at different anatomic locations. Some, such as near-infrared, EEG, and MEG sensors, are positioned on or above the scalp, while others penetrate the body to differing extents, including electrocorticography (ECoG) and local field potential (LFP) sensors.

3.4.1. Data Processing in a Hybrid Setup

Data preprocessing is essential for minimizing noise and enhancing the quality of the integrated signals. For MUA and SUA, preprocessing steps include spike sorting, artifact removal, and filtering to isolate neural activity from background noise. MEA data often require denoising and correction for drift or signal attenuation. ECoG data, which may be affected by physiological artifacts such as muscle activity or electrical interference, require filtering and artifact rejection. The preprocessing phase ensures that the signals are cleaned and standardized for subsequent analysis.

Feature extraction involves identifying relevant neural features from each modality. For MUA and SUA, features such as the spike rates, firing patterns, and interspike intervals are extracted. MEA provides data on local field potentials (LFPs), which reflect population-level activity [91]. ECoG signals are analyzed for broadband oscillatory activity and event-related potentials. Integrating features from these diverse sources involves aligning the spatial and temporal domains, creating a unified dataset that captures both individual neuron activity and broader cortical dynamics.

Advanced analytical techniques are employed to fuse and interpret data from MUA/SUA, MEA, and ECoG. Methods such as cross-correlation, coherence analysis, and dimensionality reduction are used to explore relationships between single-unit and population-level activity. Machine learning algorithms and statistical models facilitate the integration of complex datasets, enabling researchers to identify patterns and interactions across different scales of neural activity [92]. Visualization tools are also utilized to represent the combined data, providing insights into the functional connectivity and spatial organization of neural networks.

3.4.2. Challenges in a Hybrid Setup

A significant challenge in integrating MUA/SUA with MEA and ECoG is the disparity in spatial and temporal resolutions. MUA and SUA provide high-resolution data at the level of individual neurons, whereas MEA offers a broader view with moderate resolution, and ECoG captures the activity from larger cortical areas. Aligning these data sources requires careful consideration of their respective resolution capabilities to avoid potential mismatches in spatial localization and temporal timing.

The complexity of integrating data from multiple sources is a major challenge. Combining MUA, SUA, MEA, and ECoG data involves sophisticated data processing and synchronization techniques. The need to align signals from different spatial scales and coordinate various preprocessing steps can complicate the integration process. Advanced computational methods and robust data-handling protocols are necessary to manage and interpret the combined datasets effectively.

Artifact management is crucial when dealing with multiple recording modalities. Each method has its own sources of noise and interference, which can affect the quality of the integrated data. Ensuring effective artifact removal and noise reduction is essential to maintain the accuracy and reliability of the results. The presence of artifacts can obscure meaningful neural signals and hinder the integration process.

The practical aspects of setting up and maintaining a hybrid MUA/SUA-MEA/ECoG system pose additional challenges. Coordinating the use of different equipment, ensuring compatibility, and managing the physical setup can be complex. The integration process must accommodate the requirements of each modality while maintaining participant comfort and minimizing potential interference between recording systems.

3.5. fMRI-MEA/ECoG-EEG Integration

The integration of fMRI with MEA and ECoG combines the strengths of high-resolution spatial imaging from fMRI with the detailed temporal and spatial activity insights provided by MEA and ECoG. This hybrid approach enables a comprehensive analysis of brain function, enhancing our understanding of neural dynamics across different scales. The following subsections explore the data-processing methods and challenges associated with fMRI-MEA/ECoG integration.

Aligning data from these modalities necessitates meticulous synchronization, as each operates with distinct temporal and spatial resolutions. fMRI provides high-resolution spatial imaging of brain activity based on the BOLD (Blood Oxygen Level-Dependent) signal, but it is temporally limited. In contrast, MEA and ECoG capture rapid, localized cortical responses with high temporal precision. To synchronize these streams, event markers or triggers are often applied to coordinate data capture across modalities. Solutions for achieving this integration can include hardware setups for aligned signal acquisition or software tools designed for consistent timestamping. This precise alignment ensures an accurate interpretation of the neural events observed. Figure 21 illustrates an example setup for integrating fMRI with EEG, demonstrating the interaction of fMRI with electrical recordings. This multimodal strategy enhances the fidelity of neural data interpretation, contributing valuable insights into the complexities of brain function by bridging different scales and temporal frameworks of neural activity.

This process leverages an MR receiver coil to concurrently record fMRI and evoked potential (EP) signals. Amplified and digitized EP data are modulated at distinct frequencies by an MR link recorder, making it visible to the MR receiver coil. The MRI scanner’s receiver coil detects both the RF echo near the MRI center frequency and the spectrally isolated EP data, which are embedded into the extended field of view of the reconstructed MR image.

3.5.1. Data Processing in a Hybrid Setup

Preprocessing is essential to clean and standardize data from fMRI, MEA, and ECoG. fMRI data preprocessing includes motion correction, spatial normalization, and temporal filtering to remove noise and artifacts. MEA data require spike sorting, noise filtering, and drift correction, while ECoG data need artifact removal from physiological sources such as muscle activity. Combining these modalities involves aligning the preprocessed data to account for differences in spatial resolution and noise characteristics, ensuring that the integrated dataset reflects accurate neural and hemodynamic information.

Feature extraction involves identifying relevant metrics from each modality. For fMRI, features include BOLD signal changes and activation maps. MEA data are analyzed for spike rates, firing patterns, and local field potentials (LFPs), while ECoG provides information on oscillatory activity and event-related potentials. Integrating features from these sources requires aligning spatial coordinates and temporal events. Techniques such as spatial normalization, feature concatenation, and multimodal fusion are used to create a unified representation of brain activity that combines the spatial detail of fMRI with the temporal precision of MEA and ECoG.

The integration of fMRI with MEA and ECoG can be used for real-time applications such as neurofeedback and brain–computer interfaces (BCIs). The real-time processing of fMRI, MEA, and ECoG data allows for the dynamic monitoring and modulation of brain activity, enabling feedback mechanisms based on neural and hemodynamic states. This integration supports advanced applications in cognitive training, rehabilitation, and personalized neuromodulation [94].

3.5.2. Challenges in a Hybrid Setup

A major challenge in integrating fMRI with MEA and ECoG is the discrepancy in spatial and temporal resolutions. fMRI provides high spatial resolution but has limited temporal resolution, while MEA and ECoG offer high temporal resolution but with different spatial scales. Aligning these data sources requires careful consideration of their resolution differences to avoid mismatches in spatial localization and temporal synchronization.

Combining fMRI, MEA, and ECoG data involves complex data processing and integration procedures. The need to align signals from different spatial and temporal domains requires sophisticated computational methods and robust data-handling protocols. The integration process must address issues related to data fusion, feature alignment, and dimensionality reduction to create a cohesive dataset.

Managing artifacts is a significant challenge when integrating data from multiple recording modalities. Each modality is susceptible to different types of artifacts, such as motion artifacts in fMRI and electrical noise in MEA and ECoG. Effective artifact removal and noise reduction techniques are necessary to ensure the quality and reliability of the integrated data. The presence of artifacts can obscure meaningful signals and complicate the integration process.

Handling the large volumes of data generated by fMRI, MEA, and ECoG requires substantial computational resources and storage capacity. The integration of high-resolution signals from multiple sources necessitates efficient data management strategies and advanced analytical tools. Researchers must ensure that their infrastructure can support the processing and storage demands of comprehensive datasets.

3.6. fMRI/ECoG-MEG/EEG Integration

Integrating ECoG with MEG or EEG offers a multi-faceted approach to studying brain activity. This hybrid setup leverages the high spatial resolution of ECoG with the superior temporal resolution of MEG/EEG, facilitating a comprehensive understanding of neural dynamics. Below, we delve into the data-processing methods and challenges associated with ECoG-MEG/EEG integration.

Synchronizing data acquisition from ECoG and MEG/EEG is crucial for accurate integration. ECoG records cortical activity directly from the brain’s surface, providing high spatial and temporal resolutions, whereas MEG and EEG capture electrical activity from the scalp with varying spatial resolutions. To achieve temporal alignment, synchronized triggers are typically deployed, enabling coordinated data collection across ECoG and MEG/EEG. Recording systems may be linked through dedicated software tools that establish consistent timestamps and enhance data coherence. Temporal alignment can be further reinforced by using time-locked tasks or stimuli, which serve as common reference points across recordings. This integration of spatial and temporal strengths is particularly beneficial for applications requiring detailed analysis of rapid neural processes across specific cortical regions. This may involve using synchronized recording systems or software tools to ensure consistent timestamps. Furthermore, advanced data-processing techniques, such as co-registration algorithms, help in accurately aligning ECoG and MEG/EEG data spatially, ensuring that signals from corresponding brain regions are matched. The enhanced spatiotemporal resolution achieved through this integration holds promise for advanced neuroimaging applications, such as real-time brain–computer interfaces and studies of connectivity networks in cognitive and motor functions. Figure 22 illustrates the workflow for an fMRI-MEG/EEG hybrid setup, highlighting the critical steps in data acquisition, synchronization, and analysis. Figure 22 illustrates the process of deriving EEG/MEG source connectivity through a hybrid fMRI-MEG/EEG setup. The pipeline involves source imaging EEG/MEG signals using a head model built from a template or patient-specific MRI. Neuronal activity within brain regions of interest is then analyzed over time to establish connectivity patterns in the source space.

3.6.1. Data Processing in a Hybrid Setup

The hybrid setup of ECoG-MEG/EEG can be applied in real-time scenarios such as neurofeedback and brain–computer interfaces (BCIs). Real-time analysis of integrated data allows for the dynamic monitoring and modulation of brain activity. This integration supports advanced applications in cognitive training, brain state classification, and personalized neuromodulation, providing immediate feedback based on neural dynamics.

Feature extraction from ECoG includes identifying neural oscillations, event-related potentials (ERPs), and local field potentials (LFPs). MEG and EEG data provide complementary features such as oscillatory patterns and spectral power. Integrating features involves aligning spatial coordinates and temporal events, which can be achieved through methods such as spatial normalization and temporal interpolation. Techniques like multimodal fusion and concatenation are employed to combine these features, resulting in a unified representation of brain activity [96].

Preprocessing involves cleaning and standardizing data from both ECoG and MEG/EEG. For ECoG, preprocessing includes removing noise from electrical interference and correcting for signal drift. MEG and EEG preprocessing typically involves filtering to remove artifacts from muscle activity, eye movements, and other sources of noise. Both ECoG and MEG/EEG data are filtered to remove irrelevant frequencies and enhance the signals of interest. Artifacts common to ECoG and MEG/EEG need to be addressed through advanced filtering techniques and artifact correction algorithms.

3.6.2. Challenges in a Hybrid Setup

One of the primary challenges in integrating ECoG with MEG/EEG is the discrepancy in spatial and temporal resolutions. ECoG provides high spatial resolution and precise temporal data from the cortical surface, whereas MEG/EEG offers broader spatial coverage but with lower resolution. Aligning these different data types requires addressing issues related to spatial localization and temporal synchronization to ensure an accurate representation of brain activity.

The integration of ECoG with MEG/EEG involves complex data processing and fusion. Combining high-resolution signals from ECoG with broader, less localized signals from MEG/EEG requires advanced computational methods. Ensuring that the data from both modalities are accurately aligned and integrated poses significant technical challenges, necessitating robust algorithms and processing techniques.

Artifacts can significantly impact the quality of both ECoG and MEG/EEG data. ECoG data may be affected by electrical noise from external sources, while MEG/EEG signals can be contaminated by muscle activity, eye movements, and other physiological artifacts. Effective artifact removal and noise reduction are critical to ensuring the accuracy and reliability of the integrated data. This requires sophisticated preprocessing techniques and careful management of potential sources of interference.

Handling the large volumes of data generated by ECoG and MEG/EEG requires substantial computational resources and storage capacity. The integration process must manage the high-resolution data from ECoG alongside the broader datasets from MEG/EEG. Efficient data management strategies and advanced analytical tools are necessary to process and analyze the combined dataset effectively.

However, while such advancements underscore the potential of integrating diverse neuroimaging techniques, it is equally critical to systematically understand the methodologies and findings across different signal types. This facilitates not only the assessment of current BCI capabilities but also informs the direction for future innovation.

3.7. Comparative Analysis of Multimodal BCI Methodologies

The table below (

Table 1. Summary of multimodal BCI studies by signal type.

Signal Type	Methodology	Key Findings	Strengths	Limitations	Notable Papers
EEG	Time–frequency analysis, machine learning (e.g., SVM and CNN)	EEG captures cortical activity and is widely used for motor and cognitive tasks; suitable for real-time BCI applications	High temporal resolution; non-invasive and portable	Limited spatial resolution; susceptible to artifacts	[24,25,42,46,48]
fNIRS	Hemodynamic response analysis, feature extraction using PCA and ICA	Measures blood oxygenation changes, effective for assessing cognitive load and emotion states in combination with EEG	Good spatial resolution; complements EEG by providing slow neural dynamics	Lower temporal resolution; sensitive to head motion artifacts	[73,74,76,77]
ECoG	Spatial filtering, classification with deep learning	High spatial and temporal resolutions for direct cortical recording, useful in motor imagery-based BCI	Excellent signal quality; suitable for complex pattern decoding	Invasive procedure; ethical considerations	[16,18,21]
MEG	Source localization, beamforming techniques	Provides insight into deep cortical structures; used in sensory and cognitive BCIs	High spatial and temporal resolutions; good for mapping brain activity	High cost; non-portable setup	[35,36,37]
fMRI	BOLD signal analysis, functional connectivity mapping	Useful for long-term cognitive state monitoring; maps brain regions activated during tasks	Excellent spatial resolution	Low temporal resolution; not real time; expensive	[51,68,69]

) systematically categorizes recent studies on multimodal BCI applications based on the signal types utilized. Each study is summarized with key details, including the specific signal type, associated data-processing techniques, and targeted cognitive or motor functions. By presenting this information in a structured format, the table enables a comprehensive comparison of varied approaches, highlighting the trends, strengths, and potential limitations inherent in different methodologies.

This organizational approach facilitates a deeper, more nuanced understanding of how various BCI techniques have been applied across the field. By examining these applications in a structured manner, it not only highlights the current state of BCI research but also aids in identifying promising directions for future exploration and development. This analysis underscores the critical role that specific neural signals, whether cortical, subcortical, or other neurophysiological signals, play in the effective development and refinement of multimodal BCI systems. By integrating different types of neural signals, these systems can achieve levels of performance and accuracy that were previously unattainable. Furthermore, this combination of signals expands the potential applicability of BCIs, making them more versatile and capable of addressing a broader range of needs, from healthcare applications to human–computer interaction. The ability to harness and process multiple neural signals in tandem represents a significant leap forward in the field, facilitating the creation of more sophisticated, reliable, and user-friendly BCI systems that can serve a wider array of users and applications.

4. Discussion

The integration of diverse brain-imaging modalities, such as fNIRS, EEG, MEG, ECoG, fMRI, MEA, and MUA, substantially expands our capacity to capture the complexities of brain activity. Each modality provides unique insights into brain function through distinct spatial, temporal, and invasiveness profiles, which, when combined, yield a more nuanced understanding of neural dynamics. The use of multimodal approaches enables a more robust and precise analysis of the spatial and temporal characteristics of brain signals, enhancing BCI performance and reliability.

Integrating multiple modalities can improve the accuracy and precision of BCI performance. Moreover, potential BCI users, applications, validation, getting BCIs to users, the roles of government and industry, plasticity, and ethics have also been explored [97]. Different modalities can be used together to provide both spatial and temporal information about brain activity [98]. fNIRS has high temporal resolution (in the order of milliseconds) and can be used to measure brain activity in real time [99]. However, fNIRS has limited spatial resolution, and it is susceptible to motion artifacts and noise from systemic physiological activity. fMRI has high spatial resolution but limited temporal resolution. On the other hand, ECoG provides excellent spatial and temporal resolutions by recording directly from the cortical surface. It allows for the precise localization of brain activity but is invasive and limited to clinical settings. MEA and MUA further complement these modalities by offering detailed insights into neural activity at the microelectrode level. MEA records activity from multiple neurons simultaneously, providing high spatial and temporal resolution data. MUA, on the other hand, captures the combined activity of multiple neurons within a localized area, offering insights into neural population dynamics.

In contrast, fMRI offers exceptional spatial resolution capable of differentiating fine anatomical details, yet its limited temporal resolution restricts its ability to track rapid neural oscillations. This constraint can make it less suitable for applications requiring instantaneous response feedback, such as closed-loop BCIs. Nevertheless, the inclusion of fMRI in multimodal configurations can provide essential data regarding brain connectivity and network organization, which are critical for mapping long-term neural plasticity and adaptive learning processes in BCIs.

ECoG stands out as a hybrid modality that bridges some of these limitations by recording directly from the cortical surface, delivering both high spatial and temporal resolutions. This dual capability enables the precise localization of neural responses and rapid signal capture, making it invaluable for both research and clinical applications in BCIs. However, ECoG’s invasive nature restricts its use primarily to clinical settings where the risks are justified by medical necessity. Nevertheless, ECoG’s high precision in mapping language, motor, and sensory areas has demonstrated transformative potential in BCI systems for patients with severe motor disabilities.

At the microelectrode level, MEA and MUA allow for an even finer granularity of neural recording. MEA captures activities from multiple neurons simultaneously, providing rich spatial and temporal data on localized neural networks, while MUA aggregates the firing activity of neuron populations, which can reveal population dynamics not observable at the single-neuron level. These methodologies are particularly effective in experimental settings, offering an unprecedented view of neural interactions and connectivity patterns. When combined with non-invasive modalities, MEA and MUA data can be extrapolated to inform broader BCI applications and training protocols by identifying baseline neural patterns and decoding neuronal activity with a precision that could enhance BCI accuracy and functionality.

In sum, the convergence of these imaging modalities in a BCI framework not only improves signal reliability and robustness but also fosters a holistic view of brain activity that single modalities cannot provide. This multimodal approach supports adaptive BCIs that leverage both real-time and contextually relevant signals, paving the way for more nuanced interpretations of brain dynamics. Future developments in BCI will likely depend on continued advances in multimodal data fusion algorithms, which will optimize signal integration and minimize noise, ultimately enhancing BCI adaptability and user control.

5. Conclusions

To conclude, the field of brain–computer interfaces (BCIs) is poised at the intersection of remarkable technological innovation and extensive practical application. This study underscores the wide-reaching potential of BCIs across clinical, assistive, and entertainment domains, underscoring their capacity to restore lost motor functions, enhance cognitive capabilities, and redefine interactive experiences. The distinct spatial and temporal resolutions of each imaging modality enable a layered and nuanced understanding of brain activity, especially when combined within a multimodal framework of hybrid modalities, particularly EEG-fNIRS, illustrating their strengths to elevate BCI precision, reduce susceptibility to noise and artifacts, and improve overall resolution. Each modality, from fNIRS with its real-time temporal insights to MEG and fMRI with their high spatial resolution, plays a vital role in advancing our understanding of neural processes. ECoG and invasive microelectrode techniques further refine this picture, allowing access to single-unit activities and capturing fine-grained neural dynamics with high spatial and temporal fidelity.

The canvas of BCI applications extends far and wide, encompassing clinical interventions, assistive technologies, and immersive gaming and entertainment domains. These applications epitomize the versatility of BCIs, from restoring motor function in paralysis patients to augmenting cognitive abilities in the neurotypical populace. BCIs also stand poised to revolutionize interactive experiences in the realm of gaming and entertainment. Each modality has different spatial and temporal resolutions, which can be combined to provide a more comprehensive picture of brain activity [100]. We have explored various hybrid modalities in their earlier studies [101,102,103,104,105,106] out of which EEG-fNIRS stands out [107,108,109,110]. This review has outlined the myriad merits stemming from multimodal integration, encompassing heightened spatial and temporal resolutions, diminished susceptibility to noise and artifacts, and augmented precision. The integration of fNIRS, EEG, fMRI, MEG, ECoG, and MUA/SUA modalities provides a powerful toolkit for studying brain activity from multiple perspectives. By combining the strengths and addressing the limitations of each modality, researchers can achieve a more comprehensive and accurate understanding of neural processes. This multimodal approach not only enhances spatial and temporal resolutions but also improves the ability to study complex brain dynamics and connectivity.

The synthesis of multiple imaging modalities—fNIRS, EEG, MEG, fMRI, ECoG, MUA, and SUA—equips researchers with an unparalleled toolkit for examining brain function, dynamics, and connectivity. This multimodal approach not only refines spatial and temporal resolutions but also offers a more comprehensive lens through which complex neural interactions and brain connectivity can be understood. The adaptability of this framework to various research needs makes it a cornerstone in both basic neuroscience and applied BCI research, providing essential insights for studying disorders, creating prosthetics, and developing immersive virtual environments. As technology advances, the potential for multimodal BCIs will continue to grow, offering new opportunities for both research and clinical applications. By unveiling the benefits and complexities of multimodal fusion and illustrating diverse application scenarios, this paper contributes to the nascent narrative of BCIs. As the trajectory of BCIs continues to evolve, future research endeavors should pivot towards refining multimodal integration methods and exploring new horizons for advanced BCIs across diverse domains.

As BCI technology evolves, the demand for advanced multimodal integration methods and sophisticated data fusion algorithms grows. Future research should emphasize the optimization of these algorithms, aiming to minimize computational complexity while maximizing data coherence. Continued innovation in this direction promises to unlock new realms of BCI functionality, facilitating seamless interactions with technology and enhancing quality of life across diverse applications. This paper, thus, contributes to a growing body of research that shapes the ongoing narrative of BCI technology. Through refined multimodal integration and expanded exploration, BCIs are set to redefine neurotechnology, opening doors to groundbreaking applications in rehabilitation, cognitive enhancement, and beyond. The trajectory of BCI research is one of immense potential, and by charting these advances, this study lays the groundwork for further developments, encouraging future researchers to push the boundaries of BCI capabilities across clinical, neuroenhancement, and experiential domains.