Fluorescence-Based Neurotransmitter Detection: Nanomaterial Engineering and Bioanalytical Advances at the Nano–Neuro Interface

Abstract

All forms of neural communications, from cognition to emotion, are regulated by neurotransmitters, which are otherwise the chemical language of the brain. Precise detection of these neurotransmitters is essential for the perception of neurophysiology and diagnosis of neurodegenerative diseases as well. Among the existing techniques for the detection of these molecules, fluorescence sensing is evolving as a powerful approach in terms of high sensitivity, rapid response, and real-time visualization of the chemical events occurring in the neural system. In recent years, nanomaterials have transformed this field by integrating tunable optical properties, excellent photostability, and modifiable surface chemistry into biocompatible nanostructures. We summarize the recent advances of these architectures to show how the material type and dimensionality, as well as the surface functionality, play roles in sensing through the mechanisms of Förster resonance energy transfer (FRET), photoinduced electron transfer (PET), inner filter effect (IFE), and aggregation-induced emission (AIE). The discussion has also been extended to the correlation of fluorescence modulation with the selectivity and sensitivity in the mechanism-to-function relationship. The potential utility of such innovative technologies, including artificial intelligence, spectral deconvolution analysis via big data algorithms, and chip-integrated sensing, was explored as a means to enable real-time neurochemical detection. This converging area of nanotechnology and neuroscience leaves a mark not just in analytical accuracy, but also parallels human brain rhythms.

1. Introduction

The complexity of the human nervous system is controlled by complex chemical language of neurotransmitters, which are low-molecular-weight cell signaling biomolecules. These molecules are key elements in the control of synaptic communication [1]. They are critical in the regulation of all neural function, sensation, motor control, affective behavior, and endocrine control. Neurotransmitters function on target cells through selective receptors. Consequently, the resulting cellular response and the overall balance within neural signaling networks are dynamic processes [2]. Slight changes in the concentration of certain neurotransmitters will result in very adverse health conditions such as Parkinson’s disease, schizophrenia, Alzheimer’s disease, epilepsy, depression, and anxiety disorders [3]. Neurotransmitter release occurs at very low levels, typically within a range of nanomolar to micromolar and within highly confined synaptic spaces (spatiotemporal scales from milliseconds to seconds). Due to their short lifetimes, it remains a challenge to achieve real-time monitoring at high resolution across spatial, temporal, and chemical domains simultaneously [4]. To make matters worse, a number of neurotransmitters have quite similar structural families, especially in the monoamine family (like dopamine, norepinephrine, and serotonin. That causes difficulty in selective detection in the complex biological matrices, including cerebrospinal fluid, brain slices, and living tissue [5]. All of these restrictions lead to a pressing demand for sensitive detection methods. Among traditional methods, electrochemical sensing has been utilized largely due to its sensitivity and speed [6]. Techniques such as amperometry, cyclic voltammetry, and differential pulse voltammetry have been successfully employed to detect neurotransmitters like dopamine and serotonin based on their redox behavior (Figure 1) [7]. More recently, field-effect transistor (FET)-based sensors incorporating nanomaterials have shown promising improvements in detection limits and miniaturization [8]. However, these methods have critical limitations and drawbacks, such as poor selectivity due to overlapping redox potentials, susceptibility to electrode fouling, biofouling in vivo, and the difficulty of multiplexed detection in real biological environments. In addition, most electrochemical platforms are either invasive or difficult to integrate with real-time, non-invasive monitoring tools [9].

Figure 1. Schematic comparison of conventional neurotransmitter detection methods with fluorescence-based sensing approaches.

Positron emission tomography (PET), magnetic resonance spectroscopy (MRS), and fluorescence microscopy are powerful alternatives that are imaging-based approaches. PET and MRS provide in vivo neurochemical imaging with reasonable spatial resolution; however, they are limited by poor temporal resolution, high operational costs, the requirement for radiolabelled tracers, and limited chemical specificity [10]. Optical techniques using genetically encoded sensors or fluorescent reporters are being used for tracking neurotransmitters with high spatiotemporal precision, especially in living cells and small organisms [11]. Conventional fluorophores are often photo-bleachable, environmentally sensitive, and limited in spectral range [12]. Also, some require genetic manipulation or protein engineering that restricts their clinical applicability. Microdialysis and chromatographic approaches, such as high-performance liquid chromatography (HPLC) with fluorescence or electrochemical detection, exist as the gold standard for quantifying neurotransmitters in biofluids [13]. However, these methods are fundamentally offline, time-consuming, and are incompatible with dynamic or real-time measurements. Furthermore, the high costs, labor intensity, and complexity of sample preparation limit their use to highly controlled laboratory environments [14]. Sensing platforms that combine high sensitivity, rapid response, molecular selectivity, photostability, and compatibility with complex biological media are needed to overcome the aforementioned limitations. Fluorescence-based detection techniques stand out for neurotransmitter sensing among the existing detection methods despite certain existing limitations [15]. Fluorescence-based sensors are capable of real-time monitoring of analytes with sub-nanomolar sensitivity, rapid kinetics, and non-invasive readouts [16]. They are also suitable for miniaturization and integration into portable or implantable devices. Genetically encoded fluorescent sensors (GEFS) are emerging sensors for monitoring neurotransmitters in living systems due to their ability to target specific cell types and the subcellular level. These sensors are engineered from G protein-coupled receptors or ligand-binding proteins, which are fused to fluorescent probes. Despite the advantages provided by GEFS, there are limitations that hinder the application of these sensors on a broader scale. Many genetically encoded neurotransmitter sensors exhibit limited dynamic range or suboptimal SNR. This affects the quantitative accuracy under physiological conditions. Moreover, the slower binding and unbinding kinetics cannot perform well with the rapid release of neurotransmitters as well as clearance at the synapses [17].

In recent years, nanomaterials have revolutionized the design of fluorescent sensors by providing tunable optical properties, high photostability, and versatile surface chemistry within structures that can be engineered at the nanoscale [18]. Modification of size, shape, surface functional groups, and heteroatom doping significantly enhances sensitivity as well as selectivity, even in complex, interference-rich biological media. While significant progress has been made in the field, the literature remains fragmented [19]. The existing literature focuses narrowly on specific detection methods, single classes of nanomaterials, or broad biosensing strategies without adequate attention to fluorescence mechanisms.

The present review addresses this gap by providing a detailed examination of fluorescence-based neurotransmitter sensing. This work aims to unite information into a single framework by including carbon-based, metal-based, semiconductor, polymeric, and hybrid nanomaterials, fluorescence modulation mechanisms, and integrating a discussion of materials, mechanisms, and target analytes. Particular attention is given to how neurotransmitter structure influences sensing strategy, how surface chemistry and nanostructure design impact optical responses, and how these principles translate into real-world applications from in vitro diagnostics to in vivo neurochemical monitoring. Although these sensors exhibit high sensitivity, translation to clinical in vivo monitoring, which requires real-time monitoring of neurotransmitter release with spatiotemporal resolution, remains a limitation. Advancing trends such as artificial intelligence (AI) for fluorescence signal analysis, integration with microfluidics and lab-on-chip devices, and the development of wearable and implantable optical probes are also studied.

2. Classification of Nanomaterials for Fluorescent Neurotransmitter Sensing

Nanomaterials in fluorescence-based neurotransmitter sensing are broadly classified into carbon-based, metal and metal oxide, semiconductor, polymeric/organic, and hybrid/composite categories, as shown in Figure 2.

Figure 2. Schematic overview of nanomaterials and their mechanism in neurotransmitter sensing.

Each class provides unique optical and physicochemical properties, which occur through specific fluorescence mechanisms [20]. The selection of an appropriate nanomaterial is performed not only by its photophysical characteristics but also by its compatibility with the target neurotransmitter’s chemical structure and the targeted sensing environment. The analytical performance metrics, such as detection limit, selectivity, temporal resolution, and operational stability, are also being taken into consideration (Table 1) [21]. Table 1 shows the performance trends between nanomaterial platforms, in which carbon-based systems provide sensitivity along with biocompatibility, and the metal nanoparticles (NPs) class achieves lower detection limits with stability.

Table 1. Representative nanomaterial-based fluorescent sensors in neurotransmitter sensing.

Nanomaterial Type	Sensor	Target Neurotransmitter	Sample	LOD	Linear Range	Ref.
Metal nanoparticles	Gold nanoflower/Tb3+	Dopamine	Serum	0.21 nM	0.8–300 nM	[22]
Quantum dots	Ab-CdSe/ZnS-QDs	Cortisone	Saliva	100 pM	0.4–400 nM	[23]
Metal nanoparticles	Copper nanoparticles	Adrenaline	Artificial urine	2 × 10−7 M	3 × 10−5–5 × 10−7 M	[24]
Carbon-based nanomaterials	Polypyrrole/graphene quantum dots	Catecholamine (dopamine, epinephrine, and norepinephrine)	Human serum	0.007, 0.7, 5 µM	0–250, 0–400, and 0–500 µM	[25]
Carbon-based nanomaterials	Nitrogen and phosphorus co-doped carbon dots	Catecholamine (dopamine, adrenaline, and noradernaline)	Buffer	0.1 µM	0–20 µM	[26]
Metal nanoparticles	Cupric acid nanoparticles@CeO2	Dopamine	Serum	16.34 nM	0.025–98.5 µM	[27]
Quantum dots	3-mercapropionic acid-coated cadmium telluride quantum dots	Epinephrine and norepinephrine	Human serum	2.1 nM	0.005–10 µM	[28]
Quantum dots	Nitrogen-doped MXene Quantum dots	Dopamine	Aqueous solution	18 nM	20–100 nM	[29]
Quantum dots	Thioglycolic acid-capped cadmium sulphide quantum dots	Dopamine	Aqueous solution	0.68 µM	1.0–17.5 µM	[30]
Metal nanoparticles	Gelatin-coated silver NPs	Dopamine	Human serum	0.54 nM	0.80–100 and 100–1000 nM	[31]

2.1. Carbon-Based Nanomaterials

Carbon-based nanomaterials occupy a major role in fluorescent neurotransmitter sensing because they combine optical characteristics with biocompatibility and surface modifiability. Their electronic structures, dominated by π-π-conjugated domains and heteroatom-rich functional edges, exhibit emission behavior that is tuned via size, surface chemistry, and heteroatom doping [18]. The major representatives of this class include carbon dots (CDs), graphene quantum dots (GQDs), graphene oxide (GO), reduced graphene oxide (rGO), and carbon nanotubes (CNTs). Carbon dots (CDs) are quasi-spherical carbonaceous nanoparticles, typically less than 10 nm in diameter, exhibiting excitation-dependent as well as excitation-independent photoluminescence arising from a combination of core-state emission and surface-state transitions [32]. Their high aqueous solubility, photostability, and facile synthesis from both organic and inorganic precursors make them attractive for bioanalytical applications. In neurotransmitter sensing, CDs often serve as the primary fluorophore, with detection relying on fluorescence quenching or enhancement through electron transfer, redox reactions, or binding-induced aggregation [33]. Wei et al. synthesized green-emitting carbon dots (g-CDs) through a simple hydrothermal treatment of quercetin, yielding nanoparticles with strong fluorescence at 535 nm, high aqueous solubility, and excellent stability [34]. The fluorescence intensity of the g-CDs was selectively quenched by dopamine. This occurs via an interaction attributed to the oxidative transformation of dopamine-to-dopamine quinone, facilitating electron transfer between the quinone and the emissive surface states of the CDs. Under optimized conditions, the system enabled dopamine detection in the range of 2.0–40.0 μM, with a low limit of detection (LOD) of 0.52 μM. The applicability of the probe in complex biological media was validated by quantification of dopamine in human serum samples, achieving recoveries between 95.58% and 106.87%.

GQDs are composed of single- to few-layer graphene fragments with lateral dimensions typically below 100 nm, and exhibit edge- and defect-related fluorescence in addition to quantum confinement effects [35]. The abundance of oxygen-containing groups on GQD surfaces enables covalent conjugation or noncovalent interactions with recognition elements such as aptamers, peptides, or cyclodextrins [36]. In neurotransmitter sensing, GQDs have been employed in FRET platforms, where the proximity of the GQD donor to an acceptor is modulated by analyte binding, producing ratiometric fluorescence changes. The extended π-π system also facilitates strong interactions with aromatic neurotransmitters through π-π stacking, enhancing binding affinity [37].

GO and rGO are typically used as fluorescence quenchers rather than emitters, due to their broadband absorption and efficient non-radiative energy dissipation pathways. In sensing schemes, GO or rGO can act as acceptors in FRET or as quenching platforms for dye- or nanomaterial-labeled probes [38]. Upon neurotransmitter binding, the probe-quencher interaction is altered, leading to fluorescence restoration or suppression depending on the design. Their large surface areas enable multivalent interactions, while the degree of oxidation can be tuned to balance dispersibility with electronic conductivity [39].

Carbon nanotubes, in particular single-walled carbon nanotubes (SWCNTs), exhibit intrinsic photoluminescence in the near-infrared (NIR) range that is extremely sensitive to surrounding chemical environments [40]. NIR emission of SWCNTs is observed within the biological optical window (NIR-I falls in 650–900 nm and NIR-II falls in 1000–1700 nm). Light absorption and scattering by biological tissues are significantly reduced in this region, enabling deeper tissue penetration and reduced background autofluorescence compared with probes that emit in the visible region [41]. These features make SWNT-based sensors particularly useful for monitoring neurotransmitters in brain tissues. The ability to operate within this optical window provides improved signal accuracy and spatial resolution in complex neurobiological environments. Neurotransmitter-induced modulation of SWCNT fluorescence can occur via charge transfer, dielectric environment changes, or adsorption-driven perturbation of excitonic states [42,43]. Surface functionalization with DNA, peptides, or polymers exhibits selectivity toward specific neurotransmitters while preserving the nanotube’s optical properties [44]. Carbon-based nanomaterials are explicitly used for sensing catecholamines, due to their ability to be involved in redox and π-π interactions [45]. Their low cytotoxicity compared to some semiconductor nanomaterials facilitates applications in live-cell imaging and in vivo monitoring [46,47]. Batch-to-batch variability in optical properties, especially for bottom-up synthesized CDs and GQDs, and the potential for photobleaching under high-intensity excitation, are practical challenges that must be addressed through controlled synthesis and surface engineering [48].

Intra-comparison of carbon-based nanomaterials reveals that there are differences in sensing behavior with respect to dimension, structure, and surface chemistry. It has been observed that CDs primarily depend upon surface-state-based fluorescence and redox-driven PET. This makes CDs highly stable in aqueous environments, highly biocompatible, but the selectivity is limited in biological environments. In the case of GQDs, the presence of extended π-conjugation and edge effects leads to strong π-π interactions and FRET phenomena. GO and reduced GO exhibit quenching via FRET or IFE, including a sensitive turn-off sensing mechanism. CNTs, especially SWCNTs, exhibit near-IR emission, providing photostability as well as deep-tissue compatibility. The limitation is their sensitivity to environmental disruptions and precise functionalization requirements. Evaluatively, carbon-based nanomaterials show resistance to photobleaching, biocompatibility, and aqueous dispersibility. But, the changes in fluorescence depend upon the surface state-based PET or IFE, including selectivity in biological environments that have a high amount of interferents. There are also various factors affecting the sensitivity, such as batch-to-batch variability in surface chemistry and signal reproducibility.

2.2. Metal and Metal Oxide Nanomaterials

Metal and metal oxide nanomaterials exhibit photophysical and chemical properties for fluorescence-based neurotransmitter sensing [49]. The dominant mechanism of these architectures is localized surface plasmon resonance (LSPR), which is a collective oscillation of conduction electrons that generates intense, size- and shape-dependent absorption bands [50]. This phenomenon is used to quench fluorescence via energy transfer pathways such as FRET and nanometal surface energy transfer (NSET), as well as to enhance fluorescence through plasmonic field amplification in metal-enhanced fluorescence (MEF) schemes [51]. In contrast, many metal oxides exhibit defect-state emission or act as semiconducting platforms for PET, enabling direct or indirect modulation of fluorophore emission upon analyte interaction [52].

Gold nanoparticles (AuNPs) exhibit tunable LSPR across the visible-NIR range and high biocompatibility when appropriately functionalized [53]. In neurotransmitter sensing, AuNPs serve as efficient quenchers for dye- or nanomaterial-based fluorophores, in which analyte binding changes the fluorophore-nanoparticle distance, modulating energy transfer efficiency [54]. The catechol group in dopamine, for example, can coordinate to the AuNP surface, promoting aggregation or displacement of surface ligands, both of which influence plasmonic coupling and fluorescence output [55]. Peng et al. reported a colorimetric/fluorometric dual-signal assay for dopamine detection using citrate-capped AuNPs functionalized with a 4-(4-dialkylaminostyryl) pyridinium-derived fluorophore [56]. In the absence of dopamine, the yellow fluorescence of the fluorophore was efficiently quenched by the AuNPs via FRET. Introduction of dopamine induced catechol-Au coordination, triggering nanoparticle aggregation and producing two simultaneous optical responses: a visible color change from red to blue and restoration of the fluorophore’s emission. The fluorometric mode allowed quantitative detection up to 80 μM dopamine with a limit of detection of 0.29 μM, while the colorimetric mode extended the working range to 300 μM with a detection limit of 1.85 μM. The assay was validated in spiked urine samples, and recoveries were obtained between 98.2% and 106.0%, showcasing the capability of plasmon-fluorophore coupling strategies for sensitive neurotransmitter analysis in complex biological media [56].

Silver nanoparticles (AgNPs) exhibit sharper and more intense plasmon bands than AuNPs, translating into stronger electromagnetic field enhancement and greater MEF potential [57]. In carefully engineered architectures, AgNPs boost fluorophore emission by several-fold when the interfacial distance is maintained within the optimal 5–20 nm range [58]. Sivakumar et al. synthesized fluorescent AgNPs via a chemical reduction route and functionalized them with 1,4-mercaptophenyl boronic acid (MPBA) and 40-aminobenzo-18-crown-6 (ABCE) for dopamine sensing [59]. Dopamine binding induced marked fluorescence quenching by means of FRET. The probe exhibited a low detection limit of 5.39 × 10⁻⁶ M (S/N = 3) and was also found to maintain high selectivity even in the presence of excess ascorbic acid, uric acid, and various metal ions.

Metal oxide nanomaterials such as zinc oxide (ZnO) nanostructures display visible-range emission, which arises from oxygen vacancies and other defect states, along with fluorescence intensity being highly sensitive to surface adsorption processes [60]. Neurotransmitter binding modifies surface charge density or participates in redox interactions, shifting the balance between radiative and non-radiative recombination pathways. In this context, Zhao et al. designed water-soluble, luminescent ZnO QDs, which were capped with (3-aminopropyl) triethoxysilane (APTES), exhibiting strong green photoluminescence at 530 nm [61]. Dopamine caused a significant quenching of fluorescence, due to electron transfer from the ZnO QDs to oxidized dopamine-quinone species. The probe also displayed a linear response between 0.05 and 10 μM dopamine with a low detection limit of 12 nM, alongside excellent selectivity against common interferents. Application to human serum yielded quantitative recoveries of 99–110%, highlighting the potential of ZnO QD-based probes for clinically relevant dopamine detection [61]. Titanium dioxide (TiO₂), which is weakly emissive in its pristine state, can be integrated into fluorescence sensors as a photoactive scaffold that supports PET with bound analytes or surface-linked fluorophores. Wu et al. developed a fluorescence sensing system using 6 nm phosphate-modified (P-TiO₂) nanoparticles in combination with fluorescein for the detection of dopamine, L-DOPA, adrenaline, and catechol [62]. Chelation between surface Ti (IV) ions and the enediol moieties of these analytes generated strong absorption at 428 nm, enabling the P-TiO₂ NPs to act as efficient fluorescence quenchers. The system showed high selectivity, with negligible response to noradrenaline, ascorbic acid, or salicylic acid. The detection limits were 33.5 nM for dopamine, 81.8 nM for L-DOPA, 20.3 nM for adrenaline, and 92.1 nM for catechol, with S/N for all at 3. The approach was also verified through dopamine analysis in human urine, with results further confirmed by HPLC-electrochemical detection, demonstrating strong potential for diagnostic applications [62]. Iron oxide (Fe₃O₄) nanoparticles, which are inherently strong fluorophores, are incorporated into magneto-fluorescent hybrid systems in complex biological fluids [63]. For example, Xiao et al. hydrothermally synthesized Fe₃O₄- multi-walled CNTs@Hemins for fluorescence detection of dopamine [64]. These magnetic nanozymes exhibited peroxidase-like activity of Fe₃O₄ nanoparticles and, in the presence of hydrogen peroxide, catalyzed the oxidation of dopamine to its quinone form. This henceforth gave a measurable fluorescence signal through interaction with hemin-functionalization. The sensor was applied for the detection of dopamine directly in a blood sample, and a recovery rate in the range of 81.3–96.7% was observed. Plasmonic metal nanoparticles and metal oxide nanoparticles differ in sensing behavior. AuNPs provide stability and biocompatibility when compared to AgNPs and more reproducible fluorescence modulation, FRET, or plasmon-coupled mechanisms. In the case of metal oxide nanoparticles, defect-state emission is the primary one, and surface adsorption-based PET occurs, enabling sensitive detection. But these systems are susceptible to pH and nonspecific surface fouling.

2.3. Semiconductor Nanomaterials

Semiconductor nanomaterials have been used in fluorescence-based sensing for their size-tunable emission, narrow spectral bandwidth, high molar extinction coefficients, and superior photostability compared to most organic dyes [65]. Their optical properties are controlled by quantum confinement effects, where the reduction in particle size to dimensions comparable to or smaller than the exciton Bohr radius leads to discrete energy levels and a size-dependent bandgap. This tunability allows the emission wavelength to be matched precisely to the desired spectral window, including the NIR region for deep-tissue imaging applications. CdSe, CdTe, and ZnS, which belong to the class of traditional II–VI QDs, exhibit bright, stable fluorescence and can be functionalized with a variety of ligands for biocompatibility and analyte selectivity [66]. In neurotransmitter sensing, QDs are frequently employed in FRET configurations, where their emission serves as either the donor or acceptor, depending on spectral overlap with a secondary fluorophore [67]. Neurotransmitter binding events can modulate donor–acceptor separation, surface passivation, or local dielectric properties, resulting in measurable changes in fluorescence intensity or lifetime [68]. For example, catecholamine oxidation products serve as strong quenchers for QD fluorescence through PET, through the high electron mobility of QDs’ conduction bands [69]. Zhao et al. prepared MoS₂ QDs through a single-step hydrothermal synthesis and employed them as fluorescent probes for dopamine detection [70]. In an alkaline environment, dopamine was oxidized to form dopamine quinone, which interacted with the QDs and produced fluorescence quenching via both the IFE and electron transfer. This sensor exhibited a linear response to dopamine concentrations between 200 and 600 µM (R² = 0.998) with a detection limit of 4 µM. The sensor also showed high selectivity, and validation using spiked human urine samples yielded recovery rates of 95–104.6%. The 2D material acts as a broad-spectrum quencher via both FRET and PET, while the attached quantum emitters provide high-intensity fluorescence. The large π-conjugated surface of 2D nanosheets also facilitates strong π-π stacking or hydrogen bonding with neurotransmitters like dopamine, serotonin, or epinephrine, promoting efficient transduction. Perovskite quantum dots (PQDs), a more recent addition to the semiconductor nanomaterial family, offer high photoluminescence quantum yields, narrow emission peaks, and facile spectral tunability via compositional engineering (e.g., CsPbX₃, where X = Cl, Br, I). Despite their sensitivity to moisture and oxygen, PQDs have been explored for biosensing to embed in polymer matrices or coating with silica shells to enhance stability. Their high oscillator strengths and sharp emission profiles show accurate ratiometric sensing when paired with reference fluorophores. Metal sulphide QDs, such as ZnS, Mn, or CuInS₂, provide cadmium-free alternatives with reduced cytotoxicity and broader emission spectra [71]. Their defect-state emissions are being altered through surface adsorption of neurotransmitters or by PET processes involving surface-bound recognition elements [72]. Wang et al. developed a Mn-doped ZnS QD sensor, which was integrated with a silica-supported molecularly imprinted polymer (MIP) for the selective detection of serotonin [73]. Specific binding sites for serotonin were provided by the imprinted polymer, and the rebinding induced PET-based quenching of the system. LOD was calculated as 0.69 ng/mL for serotonin analysis in human serum.

Among semiconductor nanomaterials, traditional II-IV QDs are bright and exhibit narrow emission but suffer limitations due to surface oxidation, ligand stability, and leakage of metal ions. Perovskite QDs show high photoluminescence efficiency and sharp spectral features but are sensitive to humidity and ionic environments, which further limits their stability in physiological conditions despite surface passivation. Metal sulphide-based QDs showed broader emission profiles with reduced quantum yields.

In addition to conventional fluorescent nanomaterials, upconversion nanoparticles (UCNPs) exhibit anti-Stokes emission properties. Upon NIR excitation, UCNPs emit at shorter wavelengths by minimizing background autofluorescence from biological systems and thereby improving SNR, sensitivity, and detection selectivity. Rabie et al. developed a core–shell–shell-structured UCNP-based NIR biosensing platform for dopamine detection. They reported ultrasensitive detection in the picomolar range (1–10 pM) with significantly enhanced SNR due to minimized autofluorescence and deep tissue-compatible NIR excitation [74].

2.4. Polymeric and Organic Nanomaterials

Polymeric and organic nanomaterials are used as fluorescent probes for neurotransmitter sensing in accordance with their tunability in chemical composition, surface functionality, and photophysical properties. Unlike inorganic nanomaterials, polymer-based systems are synthesized from biocompatible components, which reduces toxicity concerns and enables safe in vivo or long-term use. Their fluorescence arises intrinsically from conjugated polymer backbones or extrinsically through the encapsulation of small-molecule fluorophores within a polymer matrix [75]. Conjugated polymer dots (Pdots) are particularly notable for their exceptional brightness, which develops from high absorption cross-sections and efficient intrachain energy migration toward emissive sites. By modifying the polymer’s chemical structure, emission wavelengths are precisely tuned, and functional side chains are introduced for specific neurotransmitter interactions [76]. In such a way, the doping of boronic acid moieties in Pdots resulted in covalent binding with diol group-harboring neurotransmitters, including dopamine, turning on or off fluorescence intensity or emission wavelength [77]. In the neurotransmitter sensing, the emission characteristics of fluorophores can be modified by changing their environment due to analyte–induced factors like polarity, pH, and viscosity [78]. Another platform of organic nanostructures is micellar and supramolecular assemblies, which are obtained by self-assembly of amphiphilic molecules in water. Fluorophores can be alternately included within the hydrophobic core or attached to the surface of micelles, and a neurotransmitter binding event induces disassembly or alteration in polarity, causing fluorescence modulation [79]. The dynamic reversible nature of assembly in these structures should be appropriate for real-time sensing applications in complex and dynamic biological environments. Qian et al. constructed phenylboronic acid-labeled conjugated polymer nanoparticles for fluorescent sensing of dopamine [80]. The dopamine-selective PBA groups that are pre-immobilized on the surface of the nanoparticle were able to realize specific recognition, and the fluorescence intensity was enhanced as a result of the binding reaction. The sensor showed a low limit of detection at 38.8 nM and was able to image dopamine dynamics in both living PC12 cells and zebrafish larvae brains, demonstrating it had cellular as well as in vivo approaches. Alizadeh et al. prepared Pdots by a hydrothermal approach with neutral red, urea, and trisodium citrate and obtained the QY as high as around 30% [81]. These Pdots exhibited a bright emission maximum at 435 nm and were used for dopamine detection as a fluorescent probe. The quenching mechanism of fluorescence was due to IFE in addition to static quenching. The sensor showed an ultrawide linear detection range (0.001–900 µM) with a low limit of detection of 0.28 nM, as well as selectivity against typical interfering biomolecules and ions. As an illustrative application, Pdots were used to image intracellular dopamine in PC12 cells using a droplet microfluidic platform as well, indicating their promise as highly sensitive and biocompatible nanoprobes for neurotransmitter sensing in biological systems. In this class of nanomaterials, Pdots show higher brightness and efficient energy migration, given with the fluorescence being influenced by the microenvironment’s polarity. Fluorophores of small organic molecules provide precision in structure but undergo AIQ. When micellar and self-assembled systems are taken into consideration, they provide dynamic, reversible sensing by means of binding-induced rearrangement of structure. The stability of these molecules depends upon the dilution and ionic strength.

2.5. Hybrid and Composite Nanomaterials

Hybrid and composite nanomaterials are formed by combining two or more distinct nanoscale building blocks to utilize optical, electronic, and chemical characteristics for enhanced neurotransmitter sensing performance. This is conducted by rationally integrating materials with synergistic functionalities such as plasmonic metals, semiconductors, carbon-based nanostructures, polymers, or biomolecules [82]. In fluorescence sensing, multiple signal transduction pathways are often provided simultaneously by hybrid systems. For example, when a plasmonic nanostructure, such as a Ag or Au nanoparticle, is coupled with an emissive semiconductor QD, it enables both MEF as well as FRET-based quenching at the same time. This depends on how the analyte influences the interparticle distance and the local dielectric environment. Sensitivity and dynamic range have been improved by these dual-modality sensing systems, facilitating ratiometric readouts, minimizing artifacts from probe concentration or environmental fluctuations [83]. Carbon-based hybrid platforms have been widely investigated. CDs or GQDs are combined with metallic nanoparticles, metal oxides, or covalent organic frameworks (COFs) to integrate high quantum yields, large surface areas, and tailored surface chemistry for biorecognition [84]. Two-dimensional (2D) material hybrids such as MoS₂, WS₂, or black phosphorus nanosheets decorated with QDs or polymer dots offer another route to performance enhancement. Mani et al. developed a fluorescence-based sensing platform using a MoS₂ nanohybrid, where MoS₂ QDs were uniformly distributed over MoS₂ nanosheets (MoS₂ QDNS) [85]. In alkaline medium, the probe exhibited a concentration-dependent quenching of photoluminescence upon interaction with dopamine. The quenching was attributed to the combined mechanisms of FRET and IFE. The sensor achieved a detection limit of 0.9 nM, with two distinct linear ranges spanning 2.5 nM–5.0 µM and 5.0–10.4 µM. It showed excellent selectivity even against strong interferents such as ascorbic acid and uric acid, and was successfully validated in blood samples, further demonstrating its potential for sensitive and visual dopamine detection in real biological systems. Metal–organic framework (MOF)-based hybrids are an emerging class in neurotransmitter sensing. MOFs can serve as porous scaffolds to immobilize fluorescent dyes, QDs, or carbon nanostructures, providing analyte access channels while preventing aggregation-induced quenching [86]. In situ oxidation of neurotransmitters, which leads to fluorescence modulation through generated quinone species, is facilitated by the incorporation of catalytic metal nodes (e.g., Cu²⁺, Fe³⁺) [87]. Kumar et al. synthesized chromium-based MOFs, namely Cr-IA (itaconic acid ligand) and Cr-BTC (benzene-1,2,4-tricarboxylic acid ligand), by a solvothermal route and studied their application as fluorometric probes for dopamine detection [88]. Characterization results revealed the structural integrity, crystallinity, surface area, and stability of both MOFs. Upon dopamine addition, an increase in fluorescence was observed in the materials, with Cr-IA showing 3.4-fold rise and Cr-BTC up to 5-fold increment. The detection limits were 21 nM and 41 nM in HEPES buffer, and 26 nM and 20 nM in aqueous solution for Cr-IA and Cr-BTC, respectively. High selectivity and stability across a wide pH range were demonstrated by both probes, along with reliable performance in human serum, and visible fluorescence changes suitable for real-time monitoring.

From a translational point of view, the long-term development of fluorescent neurotransmitter sensors shall be controlled by optical performance, biocompatibility, and regulatory feasibility among various classes of nanomaterials. Carbon-based nanomaterials are particularly used for wearable and clinical applications due to their low cytotoxicity, chemical stability, and ease of scalability. Semiconductor QDs and metal-based nanomaterials provide optical properties but limit long-term stability, leakage of metal ions, which further limits their use in chronic as well as implantable sensor situations. MOFs and polymer systems provide translational requirements such as mechanistic tunability, aqueous stability, and signal reproducibility. Comparison of individual nanomaterial classes shows that carbon-based and polymer systems provide better biocompatibility and operational stability, whereas metal-based and semiconductor nanomaterials provide higher optical sensitivity at the cost of long-term reliability in complex biological conditions. Hybrid and composite designs reduce these shortcomings by combining the characteristics with structural complexity and reproducibility. The properties of the nanomaterials discussed in this section influence surface functionalization strategies and determine the primary fluorescence modulation pathways, which are explored in the following section.

3. Fluorescence Modulation Pathways and Photophysical Phenomena Involved

Fluorescence is a photophysical process in which a molecule absorbs photons of higher energy (shorter wavelength) and re-emits photons of lower energy (longer wavelength). This proceeds typically within an excited-state lifetime on the order of 10⁻⁸ to 10⁻⁹ s⁻¹ [89]. Short-lived emission is highly sensitive to perturbations in the local microenvironment, making it an ideal transduction phenomenon for chemical and biological sensing. In nanomaterial-based sensors, neurotransmitter binding events modulate fluorescence intensity, wavelength, or lifetime through specific mechanisms that govern the balance between radiative and non-radiative pathways (Figure 3). These modulation processes are categorized into true excited-state fluorescence, aggregation-driven photophysical phenomena, and optical attenuated mechanisms, including FRET, PET, IFE, aggregation-induced emission (AIE), and aggregation-caused quenching (ACQ). Together with dual-mode and multiplexed strategies, they form the mechanistic basis for achieving high sensitivity and selectivity in neurotransmitter detection. Excited-state mechanisms include PET, FRET, ICT, ESIPT, and MLCT, which are directly modulated by radiative and non-radiative decay pathways following photoexcitation. In contrast, AIE and ACQ are photophysical phenomena arising from molecular or nanoscale assemblies. These are demonstrated by restricted intramolecular motion, ICT, or MLCT, depending on the material system. IFE falls under an optical attenuation process rather than an excited-state interaction [90,91,92].

Figure 3. Representation of various fluorescence mechanisms through which neurotransmitter detection takes place.

3.1. Excited-State Fluorescence Mechanisms

3.1.1. Förster Resonance Energy Transfer (FRET)

A non-radiative phenomenon in which the excited-state energy of a donor fluorophore is transferred to a nearby ground-state acceptor molecule by means of dipole–dipole coupling. The efficiency of this energy transfer depends upon many factors, including the distance separating the donor and acceptor, the degree of spectral overlap between the donor’s emission and the acceptor’s absorption, their relative dipole orientations, and the quantum yield of the donor fluorophore [93,94]. Mathematically, the transfer efficiency (E) follows an inverse sixth power of the donor–acceptor distance (r), relative to the Förster distance (R₀) (Equation (1)),

$$E=1/[1+{\left({\displaystyle \frac{r}{R_0}}\right)}^6]$$

(1)

Because of this steep distance dependence, FRET is only significant at separations between 1 and 10 nm, which makes it exceptionally useful for translating molecular binding events into optical signals.

In neurotransmitter sensing, nanomaterials provide donor–acceptor platforms in which FRET efficiency is modulated by the presence of target molecules. CDs, semiconductor QDs, and organic dyes act as donors due to their tunable emission and high quantum yields [95], while GO, metallic nanoparticles, and conjugated polymers often act as efficient acceptors [96]. Neurotransmitters modulate the donor–acceptor system by altering spatial proximity, introducing competitive interactions, and modifying electronic states that provide resonance conditions. For example, dopamine binding can disrupt π-π stacking interactions between donor and graphene oxide, restoring quenched fluorescence and generating a quantifiable off–on response [97]. Conversely, dopamine-mediated aggregation of gold nanoparticles reduces donor–acceptor distance, thereby enhancing FRET and producing an on–off response [56]. The principal strength of FRET-based probes lies in their precision at the nanometer scale and their reversible response. Dynamic and real-time monitoring of fluctuating neurotransmitter concentrations are being enabled by these systems. Adding to it, their spectral tunability facilitates multiplexed analysis, where orthogonal donor–acceptor pairs allow simultaneous detection of multiple analytes such as dopamine, serotonin, and epinephrine [98,99]. But challenges remain in minimizing nonspecific quenching and background interference in complex biological fluids, necessitating advances in surface passivation and biorecognition element integration. Kumar et al. synthesized CDs from porcine pancreatic lipase via pyrolysis and employed them as fluorescent probes for dopamine [100]. The CDs displayed strong emission that was selectively quenched in the presence of dopamine due to its oxidation to dopamine-quinone, which adsorbed on the CD surface and engaged in FRET-mediated quenching. The sensor achieved a detection limit of 20 nM, confirmed reliable performance in environmental and biological matrices, and was further validated through visual detection on TLC paper and in aqueous media. Additionally, the CDs exhibited low cytotoxicity, enabling their use for in vitro dopamine imaging. A fluorescent aptasensor for dopamine detection has been synthesized by Teniou et al. which was based on GO as a quencher and a carboxyfluorescein (FAM)-labeled aptamer as the fluorophore [101]. In the absence of dopamine, the FAM-aptamer adsorbed on the GO surface through π-π stacking, resulting in fluorescence quenching via FRET. Upon dopamine binding, a conformational change was seen in the structure of the aptamer that released it from GO, disrupting FRET and leading to fluorescence recovery. A wide linear response from 3 to 1680 nM was obtained by the probe, with an exceptionally low detection limit of 0.031 nM, and showed high selectivity against common interferents. Its performance was validated in human serum samples, demonstrating practical applicability for clinical diagnostics.

Apart from FRET, there are energy transfer processes such as Dexter energy transfer (DET) and through-bond energy transfer (TBET) that contribute to sensing neurotransmitters under specific conditions. DET is an electron exchange mechanism operating in a short range, which requires direct overlap of the orbitals and works at <1 nm distance. In the area of neurotransmitter sensing by nanomaterials, interaction between fluorophore and analyte takes place by means of surface adsorption or spatial proximity, which is beyond the aforementioned range. Hence, in this case, the contribution of DET is very minimal. In TBET, only covalent bonding pathways are observed, which are not predominant in nanomaterial-based sensing. FRET remains the primary energy transfer mechanism utilized for fluorescence modulation in neurotransmitter-sensing systems [102,103].

3.1.2. Photoinduced Electron Transfer

PET is one of the most widely used fluorescence modulation mechanisms in neurotransmitter detection, which arises when photoexcitation of a donor fluorophore is followed by electron transfer to or from a nearby acceptor species. This electron transfer provides a competing non-radiative relaxation pathway, effectively quenching fluorescence [104]. PET proceeds by means of both donor-type and acceptor-type pathways depending on the relative energy levels of the interacting species. Donor-type PET undergoes electron transfer from the excited fluorophore to an acceptor, whereas acceptor-type PET follows when an external donor transfers an electron to the excited fluorophore. In nanomaterial-based neurotransmitter sensing, both pathways contribute depending on the redox characteristics of the analyte and the electronic structure of the sensing platform. The driving force of PET depends on the relative redox potentials of the donor excited state and the acceptor, and the process is thermodynamically described by the Rehm–Weller equation [105]. When neurotransmitters such as dopamine, serotonin, or epinephrine are introduced, their redox-active catechol or indole moieties can engage directly in PET with the fluorophore or with nanomaterials functionalized as redox-active scaffolds [106]. CDs, QDs, and dye-functionalized metal oxides are frequently employed in PET-based sensing platforms due to their well-defined band structures and tunable surface states. In the presence of dopamine, for example, electron transfer can occur from the excited fluorophore into the oxidized dopamine-quinone state, resulting in efficient fluorescence quenching. Alternatively, neurotransmitters can act as reductants, donating electrons to photoexcited nanomaterials, again suppressing radiative decay. The extent of PET depends critically on the overlap between neurotransmitter oxidation potentials and the energy levels of the nanomaterial system, which allows rational tuning of probe selectivity [107]. PET in neurotransmitter detection lies in its ability to provide highly sensitive responses to even minute changes in electron transfer pathways. Unlike FRET, which is limited by the need of nanoscale proximity and spectral overlap, PET can occur through direct orbital interactions. As a result, it is not strongly affected by the distance of the donor–acceptor allowing scope for molecular design and optimization. However, this characteristic also makes PET-based sensors more vulnerable to interference from other redox-active biomolecules such as ascorbic acid and uric acid, demanding careful surface modification and selectivity strategies. Hettie & Glass developed the sensor series on a coumarin-3-aldehyde platform (aryl appended at C4) that behaves as a directly linked donor–acceptor system operating via acceptor-excited PET (a-PET) [108]. By synthesizing benzene- and thiophene-substituted derivatives, they correlated fluorescence output with the calculated E_HOMO of the pendant aryl unit and the redox potentials of the components (Rehm–Weller framework), thereby validating PET control of a turn-on response toward primary-amine neurotransmitters (glutamate, norepinephrine, dopamine). Thiophene analogues deviated from the simple PET model, defining design limits. An optimized probe, NeuroSensor 539, exhibited enhanced photophysical properties and selectively labeled norepinephrine vesicles in live chromaffin cells, confirming its biological utility [108].

Apart from dipole–dipole-mediated FRET and redox-driven PET, nanometal surface energy transfer (NSET) has been demonstrated as an efficient way for modulation of fluorescence in nanomaterial-based sensing applications. In NSET, the excess energy of the excited fluorophore is dissipated non-radiatively to metallic nanoparticles. This mechanism treats the surface of the nanometal as the acceptor. NSET shows an inverse fourth-power dependence, which is in contrast to FRET [109]. The surface energy transfer (SET) is a refined pathway of the classical NSET to accurately display energy transfer at the nanoscale. While NSET shows the energy transfer between a dipole donor and a metal nanoparticle without considering the size of the nanoparticle, SET focuses on the size of the metal nanoparticle since surface-to-volume affects optical properties. Zhu et al. designed a fluorescent probe for the detection of dopamine with aptamer-functionalized CDs and nano-graphite [110]. This system’s fluorescence modulation was demonstrated by SET. In the absence of dopamine, strong π-π interactions resulted in quenching of fluorescence via SET. The binding of dopamine led to structural alteration in the aptamer, hence restoring the fluorescence. LOD of the sensor was observed as 0.055 nM.

3.1.3. Excited-State Intramolecular Proton Transfer (ESIPT)

Excited-State Intramolecular Proton Transfer (ESIPT) is a unique excited-state process in which a proton is transferred between a donor and an acceptor site within the same molecule through intramolecular hydrogen bonding [21]. This proton migration, triggered upon photoexcitation, is typically accompanied by pronounced Stokes shifts, which significantly enhance detection efficiency by minimizing spectral overlap between excitation and emission bands. The phenomenon was first described by Weller in the 1950s during his studies on salicylic acid, marking the earliest recognition of ESIPT as a distinct fluorescence mechanism [111]. FRET and PET-based probes exhibit strong, concentration-dependent fluorescence changes. But these probes face limitations such as background absorption, IFE, and redox-active species interferants. ESIPT systems work precisely in these conditions. Because fluorophores of ESIPT show a large Stokes shift, given that the emission is separated from the excitation wavelength and reduced background interference. ESIPT also provides signal clarity in situations where a low background is more critical than sensitivity. When compared with traditional fluorophores such as fluorescein, rhodamine, ESIPT fluorophores have a large Stokes shift, approximately around 200 nm, thereby avoiding self-reabsorption and IFE [112]. 2-(20-hydroxyphenyl) benzimidazole, 2-(20-hydroxyphenyl) benzoxazole, and 2-(20-hydroxyphenyl) benzothiazole are some of the most common ESIPT fluorophores [113]. Wei et al. prepared two NIR fluorescent probes for the detection of acetylcholine activity. These probes were developed in such a way that they had the combined effects of ESIPT and ICT. The ICT process is enhanced when the probe reacts with acetylcholine, thereby producing an electron push and pull effect. Also, a unique NIR fluorescence signal was observed along with a large Stokes shift. They have also monitored the neurotransmitter under oxidative stress in the PC12 cell line as well as the zebrafish model [114].

3.1.4. Intramolecular Charge Transfer (ICT)

In ICT, photoexcitation simulates the redistribution of electron density from the donor unit to the acceptor unit within the same fluorophore or system. This mechanism is of high priority in nanosystems with donor–π–acceptor (D-π-A) structures. In these molecules, spatial separation of frontier molecular orbitals facilitates charge displacement upon excitation [115]. The relative electron-donating and withdrawing capacity of the specific moiety, the polarity of the solvent, and the microenvironment control the extent of ICT. In neurotransmitter sensing, the binding of the neurotransmitter to the probe alters ICT by direct interaction with the donor or acceptor site or by modulating the dielectric environment, thereby producing measurable shifts in the fluorescence intensity, emission wavelength, or ratiometric response. Phenylboronic acid-functionalized conjugated polymer NPs possessed D-π-A electronic structure for ICT in the excited state. When dopamine was bound, the local polarity as well as the electronic coupling around the polymer backbone was modified, which further led to the stabilization of the ICT excited state and an increase in intensity. Aggregation of electron transfer mechanisms was ruled out with the help of control experiments, which confirmed ICT induced by dopamine binding [80].

3.1.5. Metal-to-Ligand and Ligand-to-Metal Charge (MLCT/LMCT)

Metal-to-ligand charge transfer (MLCT) and ligand-to-metal charge transfer (LMCT) have been observed in fluorophores that contain metals, coordination complexes, as well as hybrid nanomaterials. In MLCT, electron transfer from metal-centered orbital to ligand-based orbital is promoted by photoexcitation, whereas in LMCT, electrons are transferred from ligand orbitals to vacant or partially filled metal orbitals. The energy and lifetime of these charge-transfer excited states are determined by the nature of the metal center, the field strength of the ligand, the coordination geometry, and the oxidation state [116]. FRET- and PET-based systems are not mostly suitable for systems such as catecholamines, which interact strongly with metal centers. In such systems, MLCT and LMCT support is provided since fluorescence modulation in these systems originates from metal–ligand- and not donor–acceptor-based mechanisms. This hence provides more direct and stable transduction. In sensing neurotransmitters, catechol-moiety containing neurotransmitters bind strongly with exposed metal centers by means of coordination and redox chemistry. It has been observed by Govindaraju et al. that BSA-Au nanoclusters demonstrated LMCT-derived emission modified by dopamine binding [117]. Selective fluorescence quenching was observed when dopamine was added to the system, in which the interaction of the catechol group with surface metal–ligand orbitals alters the LCT excited state, thereby decreasing the emission intensity with respect to concentration. Similarly, fluorescent two-dimensional metal–organic nanosheets based on lanthanide compounds have been developed by Moghzi et al. for the sensing of dopamine [118]. Ultra-thin nanosheets with europium were designed, which demonstrated strong emission by means of a metal-centered and ligand-sensitized charge transfer process. With the addition of dopamine, changes in the excited state via modulation of local coordination and electronic structure. This study revealed that the MLCT-controlled excited state of the metal–organic system acts as a sensing probe for the detection of neurotransmitters. Excited-state fluorescence mechanisms have their own limitations in complex biological environments. PET-based sensing is highly sensitive to catecholamine neurotransmitters but susceptible to redox-active interferants. FRET-based sensing systems depend on donor–acceptor distance and are vulnerable to nonspecific adsorption of biomolecules. ICT and MLCT are influenced by polarity and coordination dynamics, leading to variability of signals under physiological conditions.

Overall, the above mechanisms primarily depend on molecular-level excited-state interactions. However, fluorescence modulation in nanomaterial-based systems can also result from aggregation effects and optical attenuation, which are discussed below.

3.2. Aggregation-Driven Photophysical Phenomena

Aggregation-Induced Emission (AIE) and Aggregation-Caused Quenching (ACQ)

Aggregation phenomena represent a unique fluorescence modulation pathway in nanomaterial-based sensing. While conventional organic fluorophores often suffer from aggregation-caused quenching due to π-π stacking and exciton decay. Certain classes of luminogens, notably tetraphenylethylene (TPE)-derived AIEgens, exhibit the opposite behavior, termed AIE [119]. In AIE systems, restriction of intramolecular rotations in the aggregated state suppresses non-radiative decay channels, thereby enhancing fluorescence [120]. By contrast, ACQ is common in plasmonic and quantum dot systems, where excitonic coupling, energy migration, and non-radiative resonance processes dominate once nanoparticles cluster [121]. In neurotransmitter sensing, both AIE and ACQ have been applied as readout mechanisms. AIE-based sensors rely on analyte-triggered aggregation of luminogens or nanocomposites, where the formation of supramolecular assemblies enhances fluorescence in proportion to neurotransmitter concentration [19]. On the other hand, ACQ-based systems make use of analyte-induced nanoparticle clustering with AuNPs, AgNPs, and semiconductor QDs. In these systems, aggregation disturbs the excitonic environment, leading to spectral broadening and a decrease in emission intensity. Dopamine and related catecholamines are well-suited to such mechanisms because their catechol groups coordinate strongly with metal ions or nanoparticle surfaces, thereby exhibiting aggregation or cross-linking [122]. The duality of AIE and ACQ mechanisms exists in AIE-type “turn-on” systems, which are especially useful in minimizing false positives, while ACQ-type “turn-off” sensors benefit from high sensitivity when aggregation is well-controlled [123,124]. However, distinction must be made between true aggregation-related emission changes and artifacts such as light scattering or secondary IFE. To confirm that aggregation is the reason behind fluorescence changes, methods such as dynamic light scattering (DLS), zeta potential measurements, and TEM imaging are used [125]. Aggregation-driven photophysical behavior produces high signal contrast but relies on the colloidal stability of the system. Under physiological conditions, which contain salts, proteins, and lipids, exhibit uncontrolled aggregation leading to changes in fluorescence results and poor reproducibility. Hence, these systems often require surface engineering and stabilization.

3.3. Optical Attenuation Phenomena—Inner Filter Effect (IFE)

IFE represents an optical attenuation process in which an absorbing species within the detection path reduces either the excitation photon flux reaching the fluorophore (primary IFE) or reabsorbs the emitted photons before they reach the detector (secondary IFE). Unlike collisional or static quenching mechanisms, IFE does not require direct excited-state interactions between analyte and fluorophore but rather arises from overlap between the analyte’s absorption spectrum and the excitation or emission wavelengths of the probe [91]. The observed fluorescence intensity diminishes not because of altered fluorophore kinetics but because of radiometric losses. This can be described by the empirical correction Equation (2)

$$F_0=F_{obs}\times 10[A \left({\lambda }_{ex}\right)+{\displaystyle \frac{A\left({\lambda }_{em}\right)}{2}}]$$

(2)

where A(λ) is the absorbance at the excitation and emission wavelengths [126]. The hallmark of IFE is that time-resolved fluorescence lifetimes (τ) remain unchanged even as steady-state intensity decreases, differentiating it from dynamic quenching processes, while the proportionality between apparent quenching and absorbance validates its optical rather than kinetic origin [127]. Many neurotransmitters or their oxidation products possess strong absorption in the UV–visible range, which is relevant in IFE. Dopamine, norepinephrine, and epinephrine, which belong to the catecholamine class, undergo oxidation to quinonoid species with broad absorption between 300 and 500 nm. This directly attenuates excitation or emission of blue/green-emitting nanoprobes such as carbon dots and ZnO quantum dots [128,129]. Nanomaterial scaffolds such as graphene oxide, MoS₂ nanosheets, or aggregated gold nanoparticles also introduce broadband extinction bands that overlap with common fluorophores, along with quenching mechanisms like PET or FRET [130,131]. As a result, apparent “turn-off” responses in these systems can be strongly influenced by IFE unless carefully deconvoluted. Spectral engineering through the selection of red or near-infrared fluorophores and long-Stokes-shift reporters minimizes spectral overlap with analyte absorption [132]. Geometric and dilution-based approaches, including front-face illumination or reduced optical path length, suppress reabsorption artifacts and allow assessment of linearity upon dilution. Ratiometric referencing is also a powerful approach, incorporating an internal band unaffected by IFE so that analyte-induced attenuation can be internally normalized. Time-resolved fluorescence techniques are widely used, since IFE leaves lifetimes invariant while true quenching processes alter them [91]. Additionally, absorbance-based mathematical corrections using A (λ_ex) and A (λ_em) restore intensity values, and orthogonal controls with non-interacting absorbers of identical absorbance can be used to isolate IFE contributions [133]. In cases where dopamine oxidation or nanoparticle aggregation generates distinct absorbers, IFE can be exploited as a reliable signal transduction mode that links molecular recognition to measurable optical attenuation. However, when fluorescence quenching via PET or FRET is the intended detection mechanism, uncorrected IFE introduces significant bias into Stern-Volmer analysis, detection limit determination, and mechanistic interpretation. Therefore, rigorous neurotransmitter assays routinely report absorbance spectra alongside fluorescence data, incorporate lifetime measurements, and validate dilution or spectral shift controls to confirm that the detected signal reflects the intended modulation pathway rather than optical filtering [134]. Lin et al. engineered a “reversed IFE” dopamine assay in which citrate-stabilized AuNPs strongly absorbed the emission band of green-emitting graphene GQDs and quenched their signal; dopamine induced AuNP aggregation, diminished the AuNP absorbance overlap, and restored GQD fluorescence [135]. The sensor showed a linear response from 20 to 200 nM with an LOD of 15 nM. Complementarily, Liu et al. used green-emissive carbon dots and exploited dopamine’s alkaline self-polymerization to polydopamine; the in situ PDA introduced broadband absorption that produced IFE (with a contributory FRET component), yielding selective DA detection over 0.1–15 µM with a 37 nM LOD and successful recovery in serum [129].

3.4. Dual- and Multi-Mechanism Fluorescence Sensing Approaches

Conventional single-sensing mechanisms generally provide only one optical output, such as fluorescence quenching or enhancement, in response to a target analyte. The limitation lies in the fact that the channel can be insufficient for decoding the complex interplay of biomolecules in living systems, where cross-talk and interference are common [38]. Dual- and triple-sensing strategies address this challenge by integrating multiple fluorescence modulation pathways into a single nanoplatform, thereby providing orthogonal or complementary readouts that reinforce analytical accuracy. Dual-sensing mechanisms such as FRET-ICT, FRET-PET, TBET-PET, ICT-PET, ESIPT-AIE, ESIPT-PET, ESIPT-FRET, and ESIPT-ICT allow two independent modes of signal transduction to operate concurrently, while triple-sensing mechanisms such as PET-ICT-ESIPT and PET-ICT-FRET extend this concept further [136].

The incorporation of nanomaterials into such systems has led to the development of dual-mode fluorescence sensors, combining mostly optical and plasmonic or optical and catalytic responses. Zhang et al. combined AuNPs with fluorophore-labeled dopamine-binding aptamers [137]. In this system, dopamine recognition produces two independent readouts: a colorimetric response, where AuNP aggregation induces a visible color change from red to blue, and a fluorescence recovery response, arising from aptamer conformational switching that restores the quenched fluorophore signal. This dual-mode strategy not only eliminates false positives but also enables highly sensitive dopamine detection, achieving a LOD of 78.7 nM within a rapid assay time of 6 min. The probe confirmed specificity against structurally related interferents and proved effective in complicated biological matrices, underlining its translational potential for clinical bioanalysis. Liu et al. synthesized a fluorescent probe using unmodified magnetite nanoparticles as peroxidase-mimics [138]. These mimics were used to catalyze H₂O₂-mediated oxidation of Amplex Ultrared. The binding of catecholamines to Fe³⁺ sites on the nanoparticle surface inhibited the nanozyme activity, leading to the quenching of AUR’s fluorescence. Henceforth, the system provided (i) catalytic fluorescence generation (Fe₃O₄ + H₂O₂ + AUR → fluorescent product), (ii) analyte-induced inhibition of fluorescence by catechol coordination to Fe₃O₄, and (iii) an accompanying redox/catalytic-activity change that is used as an orthogonal validation of binding. The authors reported limits of detection of 3 nM for dopamine, L-DOPA, and norepinephrine, and 6 nM for epinephrine, and they validated the probe for dopamine determination in urine samples.

The practical relevance of all the above fluorescence modulation pathways has been compared in terms of sensitivity, reversibility, and overall applicability in the detection of neurotransmitters, as provided in Table 2 below.

Table 2. Comparative analysis of fluorescence modulation pathways in nanomaterial-based neurotransmitter sensing.

Fluorescence Pathway	Nature of the Mechanism	Sensitivity/Reversibility	Typical Nanomaterials	Sensing Applications	Ref.
PET	Excited-state charge transfer	High/often reversible	CDs, QDs, metal oxides	Redox-active neurotransmitters (dopamine, epinephrine)	[139]
FRET	Distance-dependent energy transfer	Moderate-high/reversible	QDs, GO, plasmonic NPs	Ratiometric sensing, spatial mapping	[140]
ICT	Excited-state charge redistribution	Moderate/reversible	Organic fluorophores, conjugated polymers	Polarity-sensitive, microenvironment sensing	[141]
ESIPT	Proton transfer in excited state	Moderate/reversible	Small molecule fluorophores, polymer dots	Large Stokes shift, low background	[142]
MLCT/LMCT	Metal–ligand charge transfer	High/variable	Metal nanoclusters, MOFs, coordination polymers	Coordination/redox-active neurotransmitters	[143]
AIE	Aggregation-driven phenomenon	High/often reversible	AIEgens, metal complexes, polymer dots	High signal-to-noise turn-on sensing	[144]
ACQ	Aggregation-driven phenomenon	High (turn-off)/often reversible	Metal NPs, QDs	Turn-off systems	[145]
IFE	Optical attenuation (non-excited-state)	Apparently high/reversible	Absorbing systems	Absorbance-dominated sensing	[135]

4. Neurotransmitter Targets and Material–Mechanism Relationship

Neurotransmitters are being synthesized by the neurons and stored at the end of the axons within the vesicles located. These neurotransmitters are tiny molecules that enhance, relay, and transform signals within the cells of both the central and peripheral nervous systems [146]. Figure 4 shows the fundamental process of neurotransmission, including neurotransmitter production and various classes of neurotransmitters. Neurotransmitters are released from presynaptic neurons, while separate biosensors can identify them based on the concentration and distribution of the probes, as indicated by the intensity of the fluorescence signal [147].

Figure 4. Process of neurotransmitter release and structures of three classes of neurotransmitters.

Even though all the neurotransmitters are crucial in maintaining the nervous system, a group of specific neurotransmitters undertakes the role related to known neuropathies. Hence, the existing detection methods focus on the sensing of this small group of neurotransmitters, which include catecholamines, indolamines, amino acid neurotransmitters, and others such as acetylcholine, histamine, and adenosine, and their specific mechanism involved in the detection. Table 3 categorizes fluorescence mechanism pathways by dictated neurotransmitter structures, highlighting the mechanism–structure relationship for sensing.

Table 3. Material–mechanism relationships in fluorescence-based sensing of neurotransmitters.

Class	Examples	Key Structural and Chemical Features	Sensing Mechanism
Monoamines (Catecholamines)	Dopamine, norepinephrine, epinephrine	Catechol/phenolic ring with primary amine; strong redox activity; cis-diol functionality; oxidizable to o-quinone (electron acceptor)	PET (electron transfer to quinone intermediates) IFE (oxidized catechol absorption overlap) FRET (resonance energy transfer to oxidized intermediates) Aggregation-induced modulation (dopamine polymerization/aggregation)
Amino acid neurotransmitters	Glutamate, GABA, glycine	Lacks strong redox activity; requires recognition elements, surface modifications, or enzymatic coupling; glutamate and glycine induce aggregation/fluorescence modulation; GABA detection is often enzyme-assisted	Turn-on fluorescence (suppression of non-radiative pathways) AIE (aggregation-induced emission) Enzyme-assisted PET quenching (biochemical coupling for GABA)
Diverse neurotransmitters	Acetylcholine, histamine, and adenosine	Lack catechol moieties; detection relies on enzyme-assisted catalysis, aptamer/peptide recognition, or MOF/nanocluster structural modulation	Enzyme-assisted oxidative quenching (e.g., H2O2-mediated) Ratiometric fluorescence/FRET (dual-emission probes) Aptamer/peptide recognition (binding-induced signal recovery or quenching) Binding-induced ratiometric shifts (structural modulation by ATP/adenosine binding)

4.1. Monoamine Neurotransmitters

The structure of monoamine neurotransmitters has a catechol (or phenolic) ring linked to a primary amine. This ring generates strong redox activity and cis-diol functionality [148]. The catechol unit oxidizes to form o-quinone intermediates, which act as efficient electron acceptors, enabling quenching of emissive nanomaterials via PET. This mechanism is dominant in carbon-based nanostructures where excited-state electrons from surface or defect states are transferred to dopamine-quinone, suppressing radiative recombination. Liu et al. synthesized cysteine-capped AgInZnS quantum dots (Cys-AIZS QDs) with a large Stokes shift and excellent biocompatibility, which were further functionalized with 3-aminophenylboronic acid (APBA) to yield APBA-AIZS QDs for dopamine detection [149]. The fluorescence of the QDs was significantly quenched by dopamine through PET, attributed to the oxidation of the catechol group to o-quinone and subsequent electron acceptance. The system provided a wide detection range of 1.5–900 μM with an LOD of 0.65 μM, and was successfully extended to live-cell imaging applications, confirming its potential for intracellular monitoring of neurotransmitters. Tiwari et al. reported the synthesis of CDs and nitrogen-doped CDs (NCDs) with exceptionally high quantum yields of 54.29% and 89.82%, respectively, which were applied for dopamine sensing in human serum [150]. The fluorescence of both probes was quenched efficiently by dopamine in a concentration-dependent manner, attributed to PET from the excited CDs/NCDs to oxidized quinone intermediates. The probes exhibited linear quenching trends across broad ranges of 3.3–500 μM for CDs and 3.3–400 μM for NCDs, with limits of detection of 5.54 μM and 5.12 μM, respectively. The system revealed good resistance to interference from co-existing biomolecules and enabled selective detection of dopamine in real serum matrices, underscoring the practical utility of doped carbon nanostructures for neurotransmitter monitoring.

IFE has also been extensively used for monoamine detection, since oxidized catecholamines absorb strongly in the UV–visible region and overlap with excitation/emission bands of many fluorophores. Qu et al. developed a dopamine sensor based on dual-emissive CDs (DECDs) that displayed distinct blue (385 nm) and yellow (530 nm) emission bands [151]. In this system, both emissions were efficiently quenched by AuNPs, but the introduction of dopamine triggered AuNP aggregation, thereby reducing spectral overlap and gradually restoring DECD fluorescence. This aggregation-mediated IFE allowed selective and quantitative detection of dopamine with linear ranges of 0.5–3 μM (blue channel) and 0.1–3 μM (yellow channel), and detection limits in the sub micromolar regime. The sensor was successfully applied to real samples with satisfactory recovery and further extended to evaluate tyrosinase activity, as enzymatic oxidation of dopamine into dopaquinone prevented AuNP aggregation and maintained fluorescence quenching.

FRET strategies have been applied to both dopamine and epinephrine, typically using GO or metal nanoparticles as acceptors. Wang et al. reported the construction of a cadmium-based MOF (Cd-MOF, XYNU-1) with thermal and aqueous stability that functioned as a dual-responsive biosensor for hippuric acid and epinephrine [152]. In the case of epinephrine, the probe exhibited a characteristic “turn-off” luminescence response due to energy transfer between the MOF emission and oxidized epinephrine intermediates, resulting in a ratiometric modulation of fluorescence intensity. The one-dimensional channel architecture of XYNU-1 provided abundant interaction sites, facilitating efficient analyte binding and spectral overlap required for resonance energy transfer. High sensitivity and selectivity in serum, with stable operation across a wide pH range (2–12), were obtained, and also offered simultaneous biomarker discrimination. This was done by combining ratiometric detection of hippuric acid with FRET-mediated quenching by epinephrine. Gerelkhuu et al. studied an upconversion nanoparticle (UCNP)-based sensing platform for catecholamines by decorating NaLuGdF₄:Yb³⁺/Er³⁺ UCNPs with different metal ions [153]. Surface modification with Fe³⁺, Li⁺, and Cu²⁺ provided selectivity toward catecholamines by modulating the energy transfer dynamics between the UCNP emission bands and the bound analyte–metal complexes. Under 980 nm excitation, the decorated UCNPs provided strong luminescence signals with fluorescence quenching responses that followed concentration-dependent linearity. The system achieved remarkable detection limits of 2.8 nM for catecholamines, 2.5 nM for dopamine, and 2.4 nM for epinephrine, within the 5–320 nM working range. The method was successfully applied to human urine samples, yielding high recovery and reproducibility.

In addition, aggregation-induced modulation has provided new opportunities to exploit monoamine chemistry. Zhang et al. designed a ratiometric fluorescence sensing system for dopamine based on gold nanoclusters encapsulated in hyperbranched polyethyleneimine (hPEI-Au NCs) [154]. In this study, the hPEI shell facilitated dopamine self-polymerization, which in turn induced aggregation-mediated modulation of the nanocluster emission. The Au NCs worked as both an internal fluorescence reference and a catalytic enhancer of the polymerization process by mitigating the proton sponge effect of hPEI. This synergistic design provided rapid and highly sensitive dopamine detection with a detection limit of 10 nM, along with selectivity against amino acids, sugars, and salts due to the specificity of the self-polymerization reaction.

4.2. Amino Acid Neurotransmitters

Glutamate, γ-aminobutyric acid (GABA), and glycine are essential amino acid neurotransmitters. These are the key regulators in the nervous system since they are involved in excitatory and inhibitory signaling. Imbalanced levels of these neurotransmitters are associated with neurological disorders, including epilepsy, Alzheimer’s disease, and schizophrenia [155]. Due to limited redox activity, their fluorescent detection depends more on recognition elements, surface modifications, and indirect sensing pathways. Fluorescent nanomaterials provide versatile platforms where specific functionalization enables high-affinity interactions, translating molecular recognition into quantifiable optical signals through various mechanisms [156]. Hosseini et al. observed through their study that GQDs can act as a selective fluorescent sensor for glutamate in aqueous buffered solution [157]. The GQDs exhibited a characteristic emission at ~430 nm, which was progressively enhanced upon the addition of glutamate in a concentration-dependent manner. Interaction between glutamate and GQDs inhibited non-radiative recombination pathways, resulting in increased fluorescence. The probe provided a wide linear detection range of 1.6 × 10⁻⁷ to 1.0 × 10⁻⁵ M with a detection limit of 5.2 × 10⁻⁸ M, and displayed high selectivity against other amino acids. This turn-on response represents a deviation from conventional quenching-based strategies, offering a simplified, rapid, and highly sensitive platform for glutamate detection. Dang et al. developed a copper-doped CD (Cu-CD) nanoprobe modified with calcium ions (Ca²⁺), forming a Ca@Cu-CD platform that enables AIE upon glycine binding [158]. In this system, the Cu-CDs exhibited weak baseline fluorescence, but the presence of glycine triggered strong cluster formation or aggregation, causing significant fluorescence enhancement. The probe achieved a low LOD of 17.2 nM using fluorescence and 4.1 nM via electrochemical detection. It was also validated in real human serum samples and exhibited biocompatibility, enabling ex vivo imaging in Drosophila melanogaster (fruit flies) without cytotoxicity. In contrast to glutamate and glycine, GABA requires additional biochemical coupling for reliable detection, and enzyme-assisted nanomaterial systems have therefore emerged as a powerful strategy to achieve high specificity and sensitivity. Sangubotla et al. developed an enzymatic nanomaterial-based fluorescence platform for selective GABA detection using corn-derived CDs (CCDs) functionalized with 3-aminophenylboronic acid (APBA) and NADP⁺, termed CANs [159]. The sensing mechanism relied on coupling the nanomaterial with GABase, a dual-enzyme system (GABA transaminase and succinic semialdehyde dehydrogenase), which catalyzes the conversion of GABA into succinate with concomitant reduction of NADP⁺ to NADPH. This redox transformation enabled PET-driven fluorescence quenching of the functionalized CDs, providing a quantifiable optical signal. The CAN-GABase platform exhibited a linear detection range of 0–90 μM with a limit of detection of 6.46 μM, and high selectivity against competing biomolecules. Validation in cerebrospinal fluid (CSF) and serum samples yielded recovery rates exceeding 93%, underscoring the clinical applicability of the nanosensor for inhibitory neurotransmitter monitoring. Sangubotla et al. further advanced this enzymatic nanomaterial-based approach by fabricating coffee waste-derived CDs (C-CDs) functionalized with APBA and NADP⁺, yielding C-CANs [160]. When coupled with GABase, these nanoprobes enabled selective GABA detection through the enzymatic conversion of GABA into succinate with concurrent NADP⁺ reduction, which in turn induced fluorescence quenching via electron transfer. The C-CAN-GABase platform achieved a wide linear range of 0–20 μM with low limit of detection of 95.09 nM, significantly improving sensitivity compared to earlier designs. This nanosensor was validated both in vitro and in vivo: in which intracellular GABA sensing was experimentally proven in human neuroblastoma (SH-SY5Y) cells, while zebrafish larvae studies confirmed its applicability in live organisms. Also, human serum samples were evaluated, which in turn provided excellent recovery rates of 97–105.6%, highlighting the capability of the sensor.

4.3. Diverse Neurotransmitters (Acetylcholine, Histamine, Adenosine)

Neurotransmitters such as acetylcholine, histamine, and adenosine belong to an important class of signaling molecules beyond monoamines and amino acid neurotransmitters. These are in charge of regulating processes such as synaptic plasticity, immune response, and neuromodulation in both central and peripheral systems [146]. Abnormal levels of these neurotransmitters are very much associated with neurodegenerative disorders, inflammatory diseases, and cardiovascular dysfunctions. Their structures do not have easily oxidizable catechol moieties, making their fluorescent detection more challenging. Detecting these molecules is performed by enzyme-assisted catalysis, receptor-mimicking probes, or surface-modified nanomaterials. Integrating acetylcholinesterase for acetylcholine, histamine-specific aptamers, or adenosine receptors, converting molecular interactions into quantifiable optical signals, is plausible by means of a fluorescence-based detection method. Li et al. developed a label-free and cost-effective fluorescence biosensor for acetylcholine (ACh) detection using protein-stabilized nanoclusters (NCs) as the optical transducer. A cascade enzymatic reaction was used, where acetylcholinesterase (AChE) hydrolyzed ACh to choline, followed by oxidation of choline-by-choline oxidase (ChOx) to generate hydrogen peroxide [161]. The produced H₂O₂ efficiently quenched the fluorescence of the nanoclusters through an oxidative electron transfer pathway, yielding a quantifiable decrease in emission intensity. This biosensor achieved an ultralow detection limit of 5 pM with a linear dynamic range of 0.1–20 nM, outperforming many conventional ACh assays. The system established practical applicability by accurately measuring ACh in human blood samples, with recoveries ranging between 93% and 102%, and showed strong agreement with results obtained from commercial assay kits. Similarly, Martín-Barreiro et al. described an enzymatic nanobiosensor based on the covalent attachment of gold nanoclusters (AuNCs) to the flavin adenine dinucleotide (FAD)-dependent enzyme choline oxidase (ChOx), AuNCs@ChOx, as a fluorescence sensing platform [162]. The oriented binding approach was used to stabilize the enzyme and promote structural changes that increased the endogenous FAD fluorescence signal. Choline-enzymatic oxidation proportional fluorescence changes were monitored both at 525 nm (FAD emission) and at 700 nm (AuNC emission), via a combined FRET as well as oxygen-mediated quenching. With the assistance of ACh, this ratiometric system switched to offer a false acetylcholine determination sounding up to the micromolar level (10⁻⁶–10⁻⁵ M) with satisfactory reproducibility (about 4% RSD). Unlike traditional dye-based probes, the AuNCs, as a new design, could offer a reversible, biocompatible, and chemically stable choice with the further development of a readout signal in the near-infrared region for high sensitivity. For histamine detection, nanomaterial-based fluorescence probes were also developed. Wang et al. fabricated a smartphone-based ratiometric fluorescence biosensor toward histamine detection based on molecularly imprinted blue/orange MXene QDs(MQDs), building upon a dual-emission platform [163]. Due to the highly selective molecular imprinting, combined with dual-color fluorescence readouts, quantitative and visual identification were achieved. The probe exhibited a linear relationship in the concentration range 1–60 μM, and had detection limits of around 22 nM by fluorescence measurement and ~92 nM by visual color observation. The applicability of the method in real food samples was proved, with recoveries ranging from 96.5 to 105.3%, which demonstrated its potential for monitoring histamine in practice. This study has broadened the utility of MXene QDs in bioanalysis and demonstrated a portable, field-worthy approach for fast determination of histamine. Peptide-based systems could also sense histamines. Shi et al. reported a peptide-CQD hybrid probe for histamine with high selectivity and fluorescence detection [164]. High-affinity histamine-binding synthetic peptides were isolated from a phage-displayed library, which quenched CQD fluorescence in an electron transfer-based process. In the presence of histamine, the peptides bound to the target rather than CQDs, and the fluorescence of CQDs was recovered. Of those examined, Hisp3 and its derivative Hisp3-2-C exhibited excellent selectivity for histamine even over closely related analogues. In addition, the integration of gold-coated magnetic nanoparticles (MNP@Au) enabled an extraction and detection for histamine in complex fish samples; it furnished not only sensitive low-amount determination but also excellent selectivity. This platform presented a potential approach for the detection of histamine based on hybrid biospecific interaction and nanomaterial luminescence. Adenosine and its triphosphate (ATP) have been focused on using ratiometric nanomaterials in the context of binding mediated conformational changes. Xin et al. synthesized a dual-emission ratiometric fluorescent probe (Cu NCs-Al@ZIF-90) to efficiently and sensitively sense ATP [165]. Synthesis of the probe: The detection probe was prepared by a one-step loading formation of CuNCs in ZIF-90 composite. CuNCs were AIE-active and exhibited enhanced fluorescence in the presence of Al³⁺. ATP recognition was achieved at the Zn²⁺ node of MOF, which led to probe cleavage and thus opposite fluorescence changes (CuNCs at 620 nm and imidazole-2-carboxaldehyde at 450 nm). The ratiometric approach led to the detection of ATP in 0.2–0.6 mM with a detection limit of 0.03 mM. Its facile synthesis and fast response provide potential for clinical and environmental ATP detection by this probe.

5. Strategies to Improve Sensitivity and Selectivity

Fluorescent nanomaterial-based sensors for neurotransmitter detection face several challenges, including weak signal intensity, background interference, and limited selectivity in complex biological systems. To overcome these limitations, various strategies have been developed to enhance fluorescence response, minimize nonspecific interactions, and enhance accurate detection at physiologically relevant concentrations, while selectivity is improved in specific studies. Approaches such as controlling nanomaterial size and morphology, tailoring surface functionalization with recognition ligands, employing doping or passivation to improve quantum efficiency, enhancing photostability and resistance to biofouling, and designing probes compatible with real biological matrices have significantly advanced the field, mainly by improving sensitivity and operational dependability. These strategies, demonstrated in diverse analyte-sensing platforms, provide guiding principles that can be directly adapted for neurotransmitter detection.

5.1. Nanomaterial Size and Morphology Control

The fluorescence behavior of nanoscale probes depends on dimensionality and morphological characteristics, which include quantum confinement, density of surface states, and interaction geometry with neurotransmitter molecules. CDs in the 2–5 nm range typically exhibit strong band-edge emission with blue-shifted photoluminescence, due to pronounced confinement of excitons. When the particle size increases to ~10 nm or above, surface defect states dominate, producing red-shifted or near-infrared emission that is advantageous for deep-tissue imaging and reduced autofluorescence interference [166,167]. Dopamine sensing studies have shown that smaller carbon dots (<5 nm) tend to favor dynamic quenching mechanisms through direct charge transfer, whereas larger dots with abundant oxygenated surface defects exhibit IFE-dominated responses [139]. Morphology plays an critical role in sensitivity. One-dimensional nanostructures such as ZnO nanorods or carbon nanotubes provide anisotropic charge transport pathways, enabling accelerated electron transfer between the sensing probe and neurotransmitter. This effect has been demonstrated in dopamine detection, where vertically aligned ZnO nanorods on substrates enhanced the fluorophore quenching efficiency more effectively then spherical ZnO nanoparticles [168,169]. Two-dimensional graphene oxide sheets, with high aspect ratios and extended π-conjugation, facilitate strong π–π stacking with catecholamines, allowing selective discrimination of dopamine [170]. Halawa et al. synthesized ultrasmall, spherical β-cyclodextrin-functionalized Au nanoclusters with an average diameter of 1.8 nm [171]. The molecular-like fluorescence was at 450 nm and large Stokes shift minimized background interference and improved signal-to-noise ratio. The uniform ultrasmall morphology facilitated PET between dopamine and the nanoclusters, yielding a broad linear range of 100 nM–80 μM and LOD of 20 nM. In addition, the coating enabled the clusters to exhibit supramolecular host-guest recognition, selectively sensing dopamine, excluding the other biological interferents, including ascorbic acid and uric acid. Size and morphology tuning is not only about optical modulation but also about ensuring compatibility with physiological environments. Sub-10 nm probes have higher diffusivity in extracellular matrices and show improved renal clearance, which is particularly relevant for in vivo neurotransmitter monitoring [172,173]. In contrast, anisotropic nanostructures such as nanorods or nanosheets exhibit stronger scattering and field enhancement, making them more suitable for localized signal amplification-based sensors [174]. Templated nucleation and confinement in mesoporous silica, hard templates, MOFs, or porous polymer scaffolds provide geometric constraints that reduce broad size distributions and prevent aggregation during growth of the nano system. When nanoparticles are grown inside mesoporous silica pores or within MOF cavities, the template fixes the maximum particle size and enforces narrow polydispersity [175]. Also, removing the template or integrating the fluorescent unit in the porous area also reduces non-radiative surface recombination by limiting aggregation [176]. This thereby enables consistent surface chemistry across particles. Recent reports using mesoporous silica templates being used in N-doped CDs and MOF-encapsulation strategies show improved colloidal stability and reproducible quantum yields, which in turn produce more consistent sensing calibration curves and lower variance in measured LODs across batches. These kinds of approaches are useful particularly when the sensing mechanism depends on surface-mediated PET or FRET, since the template fixes geometry and surface density of recognition moieties [177].

5.2. Surface Functionalization, Doping, and Passivation Strategies

Incorporating biorecognition elements such as aptamers, antibodies, or MIPs on the surface of the nanomaterials provides a molecular interface. This interface couples selectivity with signal transduction, thereby directly enhancing analytical performance. Introduction of heteroatoms also created abundant defect sites and electron-rich domains, which promote PET or FRET with catecholamine neurotransmitters. Also, surface passivation with oxides and polymers suppresses non-radiative recombination. ZnO-associated carbon dots demonstrate how inorganic surface functionalization improves neurotransmitter detection. Hur et al. synthesized ZnO-passivated CDs with abundant amine and carboxyl groups that displayed excitation-dependent blue emission at 440 nm [178]. The ZnO moiety acted as a catalytic interface, facilitating dopamine oxidation and enabling selective quenching of fluorescence via electron transfer. This hybrid surface chemistry exhibited a remarkably low detection limit of 1.06 nM in phosphate-buffered saline, while eliminating interference from ascorbic acid and uric acid, and demonstrated stable performance in human serum samples. Biorecognition elements, such as aptamers, also provide effective functionalization routes. Zhu et al. synthesized aptamer-labeled CDs and nano-graphite for the detection of dopamine, in which the former one was responsible for the fluorophore and chemical recognition, whereas the latter one served as an energy acceptor [110]. This worked on the mechanism that in the absence of dopamine, the CDs were adsorbed onto the surface of the nano-graphite by means of π-π stacking and hydrophobic interaction, thereby leading to the quenching of fluorescence due to NSET. In the presence of dopamine, the fluorescence was recovered when the analyte was attached to the aptamer tailored on the surface of the CDs in order to form a particular globular chain structure. The study also determined the LOD of 0.055 nM with a detection range of 0.10–5.00 nM. Similarly, Zhang et al. designed single-wall carbon nanohorns (SWCNHs) for the detection of dopamine [179]. The aptamer was bound with 5-carboxyfluorescein (FAM), serving as both the recognition element and fluorophore, while SWCNHs acted as fluorescence quenchers. The sensing mechanism relied on π-π stacking interactions. In the absence of dopamine, the FAM-aptamer adsorbed onto the SWCNH surface, resulting in fluorescence quenching. Upon addition of dopamine, the analyte selectively interacts with the aptamer, inducing the formation of a G-quadruplex structure that prevents its adsorption, thereby restoring the fluorescence signal. The system achieved a detection limit of 5 μM with a linear working range of 0.02 to 2.20 mM. MIPs coatings around fluorescent nanomaterials offer a synthetic yet highly specific recognition interface, with the added benefit of stability against denaturation and harsh environments. Khadem-Abbassi et al. embedded CdTe_0.5S_0.5/ZnS QDs within a molecularly imprinted poly (acrylamide-co-ethylene glycol dimethyl acrylate) matrix templated by dopamine [180]. After template removal, the resulting QDs@MIP system exhibited specific binding cavities that restored quantum dot photoluminescence but enabled selective quenching upon dopamine rebinding. The fluorescence response showed a linear correlation in the 2.63–26.30 µM range with a detection limit of 6.6 nM, outperforming non-imprinted controls. The polymeric shell ensured selectivity against amino acids, peptides, and other neurotransmitters, highlighting the utility of molecular imprinting for conferring analyte-specific surface functionality. Polymeric functionalization through this method introduces artificial receptor layers that enforce shape- and functional complementarity with the target neurotransmitter. Liu et al. reported a ratiometric hollow-structured CdTe@H-ZIF-8/CDs@MIPs sensor in which CdTe quantum dots provided the reference emission while CDs served as the recognition signal [181]. The hollow MIP structure amplified fluorescence via multiple light scattering and created abundant recognition sites for dopamine. Upon DA binding, the ratiometric fluorescence intensity was quenched in a concentration-dependent manner, achieving a wide linear range (0–600 nM) and a detection limit of 12.35 nM. The system offered a visual color change under UV light, demonstrated strong selectivity over structural analogues, and was validated against HPLC analysis.

6. Bioanalytical and Biomedical Applications

6.1. In Vitro Detection in Biofluids

Fluorescent nanomaterial-based sensors have been widely employed for the in vitro detection of neurotransmitters in complex biofluids such as serum, plasma, urine, and cerebrospinal fluid (Figure 5). These systems exhibit high ionic strength and protein content, which demand high selectivity and antifouling capability from sensing platforms. Halicka et al. reviewed a range of nanostructured fluorescent sensors for catecholamines and indoleamines, emphasizing their ability to achieve medically relevant detection limits in human serum and plasma [68]. Graphene oxide and carbon nanohorns, for example, act as efficient quenchers for dye-labeled aptamers, allowing dopamine quantification in spiked plasma samples through π-π interactions and fluorescence recovery mechanisms. These biofluids present three recurring analytical challenges: (i) high background absorbance and autofluorescence, (ii) abundant interferents (proteins, ascorbate, urate, metabolites) that compete chemically or optically with the target, and (iii) biomolecular fouling of the nanoprobe surface that alters binding kinetics and optical readout. Various nanomaterial systems have been engineered in such a way that they could overcome the aforementioned limitations.

Figure 5. Workflow of neurotransmitter detection with biological samples by nanomaterials.

Carbon-based fluorescent probes, such as CDs and graphene derivatives, have been applied for serum and urine assays because of their facile synthesis, high aqueous dispersibility, and modifiable surfaces. Hydrothermally generated CDs with engineered surface groups have been used to detect dopamine in serum with low- to sub-micromolar levels and recovery after increment. This thereby demonstrates that surface passivation plus appropriate sample dilution can preserve analytical performance in protein-rich fluids. Yeast-derived CDs synthesized by Cao et al. achieved a low detection limit of 30 nM and a linear range of 0.05–150 μM [182]. These CDs demonstrated how careful surface chemistry (amine/boronic acid functional groups, polymer coatings) is essential to suppress nonspecific adsorption while retaining analyte access to the recognition sites. Metal-oxide and semiconductor QDs exhibited advantages such as defect/state emission, which is useful in serum-based assays. ZnO QDs decorated with APTES, which prominently reflected the defect-state green emission, were selectively quenched by dopamine via electron transfer to oxidized quinone forms [61]. In a similar aspect, enzyme-coupled fluorescent nanoprobes remain indispensable for neurotransmitters that lack strong intrinsic redox signatures. A practical route to embed fluorophores or emissive nanomaterials within systems that report an enzyme product, such as hydrogen peroxide, quinone, or a cofactor change (NADP⁺ → NADPH). These enzyme–nanoprobe architectures mitigate selectivity concerns by converting the analyte into a unique chemical reporter that directly modulates nanoprobe fluorescence (via PET, oxidative quenching, or fluorescence enhancement) [159,183]. Wang et al. developed a simple yet powerful paper-based fluorescent test strip designed for the direct and visual detection of dopamine in minute quantities of biological fluids [184]. Dual-emission molecularly imprinted polymer nanoparticles (DE-MIPs) was designed with specific recognition sites for dopamine. These nanoparticles exhibited distinct fluorescence color changes depending on dopamine concentration, due to changes in the emission intensity ratio caused by dopamine binding and fluorescence quenching. When the DE-MIPs were coated onto a filter paper, the resulting strip functioned analogously to a pH test paper, enabling a visual, color-based dopamine readout with just a tiny drop (~10 μL) of serum. The sensor achieved semiquantitative detection within three minutes, with a detection limit in the 100–150 nM range, visible even to the naked eye. Similarly, Safwat et al. reported an ultrasensitive and rapid fluorescence-based strategy for dopamine detection using a dual-ligand terbium metal–organic framework (Tb-MOF) that functions through a turn-on fluorescence mechanism [185]. Upon dopamine interactions, Tb-MOF exhibited a significant fluorescence enhancement, offering high sensitivity and selectivity in complex biological matrices such as plasma and urine. The MOF integrates terephthalic acid to enhance Tb³⁺ luminescence and 2-methylimidazole as a selective dopamine binding site, enabling the probe to respond within 3 min of incubation under mild, physiological conditions (phosphate buffer, pH 7). This design facilitates direct and rapid quantification of dopamine after simple protein precipitation, eliminating the need for heating or lengthy preparation steps. The probe achieved a broad detection range (1 × 10⁻⁹ to 1 × 10⁻⁷ M) with an impressively low detection limit of 2.05 × 10⁻¹⁰ M.

In practical applications, sample preparation for real biological fluids requires pre-treatment in order to improve analytical precision. For blood/serum, proteins are typically removed or precipitated or centrifuged to reduce background fluorescence and protect the surface of the nanomaterial. In the case of urine samples, dilution and filtration are done to reduce the ionic strength and remove interferents before measuring the fluorescence. Microdialysis splits small and macromolecules separately which is also employed. Dopamine, norepinephrine, and DOPAC are prepared by dissolving in aqueous polyvinyl alcohol (PVA) solutions (0.25 wt%) and spin-coated onto substrates for native fluorescence [186].

6.2. Live-Cell Imaging of Neurotransmitter Release

Live-cell imaging of neurotransmitter release using fluorescence nanomaterials requires probes that combine fast kinetics (rise and decay on the order of milliseconds to seconds), high specificity, reversible binding, minimal interference from autofluorescence, and ideally, the capability for imaging in deeper tissue or complex cellular layers. Recent advances in nanomaterials have made substantial progress in fulfilling these requirements. SWCNTs or polymer- or DNA-functionalised corona phase sensors employ Corona Phase Molecular Recognition (CoPhMoRe) where noncovalent wrapping (e.g., ssDNA) provides selective molecular binding to small neuromodulators or transmitters, modulating the SWCNT’s intrinsic NIR emission (1000–1300 nm). Advantages include very low photobleaching, long emission lifetimes, deep tissue penetration, and sensitivity to catecholamines or serotonin with sub-micromolar to nanomolar detection limits. For example, the “nIRCats” family comprises (GT)₆-ssDNA on SWCNTs, giving ∆F/F up to 24-fold for dopamine in vitro, with a dynamic range spanning 10 nM to 100 µM [187]. Strategies include combining a ratiometric sensor by means of integrating two NIR signals such that changes in one sensor relative to a stable interference allow correction for motion and photobleaching. Hill et al. designed such a sensor using Egyptian blue (CaCuSi₄O₁₀) nanosheets (EB-NS) along with SWCNTs for fluorescence imaging of dopamine [188]. In this system, the SWCNT channel at 1000 nm, EB-NS at 936 nm reference was used for imaging dopamine release from differentiated Neuro 2a cells. In situ-synthesized CDs also worked as a turn-on fluorescent probe for the detection of dopamine in live cells. Tang et al. developed green fluorescence-emitting CDs through in situ from silane and dopamine [189]. In this system, dopamine reacts directly with N-[3-(Trimethoxysilyl) propyl]ethylene diamine under mild conditions to yield green-emitting SiCDs, whose fluorescence intensity increases proportionally with dopamine concentration. This turn-on mechanism allows for nanomolar-level detection of dopamine with a detection limit of 56.2 nM. The method required no sample pretreatment, demonstrated high selectivity, and performed reliably in human serum with recoveries in the range of 101.77–119.91%. Moreover, the in situ synthesis of SiCDs was also achieved within MN9D neuronal cells at physiological temperature, enabling real-time visualization of intracellular dopamine distribution. SWCNT-based nanosensor array developed by Kruss et al. enabled super-temporal resolution imaging of dopamine release from PC12 neuroprogenitor cells, obtaining a temporal precision of 100 ms per frame [169]. This sub-second imaging rate allowed real-time visualization of the onset and decay kinetics of neurotransmitter efflux following potassium ion stimulation. Each near-infrared-emissive (GA)₁₅-DNA-wrapped SWCNT acted as an individual, photostable reporter, capturing transient dopamine concentration changes in the extracellular microenvironment. The system’s high-speed acquisition effectively bridged the gap between conventional electrochemical methods with limited spatial discrimination and optical assays that lack temporal fidelity. As a result, these nanosensor arrays provided an unprecedented means to correlate the timing of dopamine release with specific subcellular regions, uncovering the dynamics of exocytosis with millisecond-scale precision. In parallel, the dense configuration of the nanosensor film afforded high spatial resolution and precise hotspot localization of dopamine efflux. Approximately 20,000 independent sensors positioned beneath a single PC12 cell generated detailed chemical maps that revealed localized “hotspots” of release, identified through amplitude fitting of fluorescence intensity traces. These hotspots were heterogeneously distributed along the cell border, often coinciding with protrusive or adhesive regions rather than strictly at protrusion tips. Statistical analyses confirmed 2–17 distinct release sites per cell, aligning with electrochemical findings while providing superior spatial discrimination. The nanosensor mapping further revealed that dopamine release was not confined to any specific curvature zone, indicating that membrane tension and adhesion likely influence release site formation [169].

To directly monitor released neurotransmitter at the cell surface (rather than relying on diffusion through the extracellular medium), probes have been anchored to cell membrane surfaces. A recent example is the DNA-nanoprism structure: a self-assembled DNA prism tethered to a membrane (via cholesterol tails) plus an aptamer module to detect dopamine [190]. Using this, transient DA release was visualized at the cell surface with high spatiotemporal resolution, and correlations were made to calcium influx triggered by depolarization (K⁺ stimulus). For physiological imaging, probes must respond reversibly (bind and release transmitter, or at least be reset) and survive repeated stimuli. SWCNT-based sensors frequently show reversible responses over repeated addition/removal of analyte in slices or extracellular space. SWCNT-DNA sensors for serotonin were derived from large libraries (10¹⁰ ssDNA sequences). These sensors exhibited high sensitivity (fluorescence enhancement up to 200%) and reversibility. They further validated for functional serotonin detection in acute brain slice studies [191].

6.3. Microfluidic & Lab-on-Chip Platforms, Wearable and Implantable Optical Sensors

Integrating fluorescence-based neurotransmitter sensing with microfluidic, wearable, or implantable systems represents a vital advancement toward translating high-sensitivity nanomaterial probes into real-time, in vivo or near-in vivo monitoring. In recent years, several studies demonstrated combinations of material innovation, optical transduction, and device engineering that begin to overcome key barriers: miniaturization, biocompatibility, signal stability, and selective delivery of probes. One recently published example is the miniaturized implantable fluorescence probe with MOFs for deep-brain dopamine sensing in rats. In this system, a fluorescent MOF specific for dopamine is immobilized on a flexible thin probe (120 µm thickness) combining thinned LEDs and phototransistors. The device operates wirelessly (0.85 g) and achieves a detection limit of 79.9 nM; applied during deep brain stimulation, it provided continuous, real-time measurement of dopamine fluctuations. The mechanical flexibility, MOF surface functionalization, and optical readout through implanted tissue illustrate how nanomaterials and electronics can be merged for deep tissue, long-term monitoring [192]. Keerthanaa et al. developed a wearable microneedle-based sensor using a biopolymer-protected Fe₃O₄-graphene oxide composite (Fe₃O₄-GO) coated with anti-fouling Nafion layers [193]. Although this example uses electrochemical readout rather than purely fluorescence, its device architecture (microneedles, flexible substrate, continuous monitoring in interstitial fluid) provides design lessons relevant for optical sensors—particularly for fluidics management, minimal invasiveness, and signal stability in physiological medium. The LOD was 90 nM via square wave voltammetry, 0.6 µM via chronoamperometry; demonstration in phantom skin gel shows potential for wearable continuous use. In the optical domain, fiber-optic and SPR sensors modified with nanomaterials offer promising implantable or semi-implantable formats. One such example is a fiber-optic SPR sensor with CuO nanoparticles of 7 nm coated over an optical fiber (gold film over unclad section) for dopamine detection. It achieves a very low detection limit of 1.11 nM, high selectivity, and a compact optical path. Such fiber-optic SPR architectures are attractive for combining small size, remote readout, and the possibility of implantation or insertion into tissue or fluidic channels [194]. Furthermore, implantable aptamer–graphene micro transistor probes have been used for in vivo monitoring of neurochemical release (dopamine) at nearly cellular spatial resolution. These devices use a soft, flexible substrate (graphene) combined with aptamer modification for neurotransmitter binding, producing optical/optoelectronic (or at least conductometric) changes that can be read in vivo following pharmacological stimulation. Sensitivity down to picomolar levels is reported, with strong dopamine vs. norepinephrine discrimination (>19-fold). Device engineering includes biocompatibility, low noise electronics, and minimizing tissue damage [195]. In designing these platforms, there are some recurring technical and mechanistic challenges to be taken into consideration such as probe material integration, optical transduction, miniaturization, flexibility, biocompatibility, antifouling, signal stability, sensitivity and selectivity in complex media. MOFs, graphene, and nanoparticles (CuO, Fe₃O₄, etc.) are being functionalized for specific neurotransmitter binding, linked with optical (fluorescence or fluorophore-MOF) or SPR/fiber-optic readout. For fluorescence, emission stability, responsive emission change (on/off, ratiometric) and possibility of excitation/emission in less scattered (e.g., near-IR) bands are essential. Implantable probes (thin LEDs, phototransistors) and wearable devices (microneedles, patches) require minimal thickness, soft substrates, wireless or low-profile interconnects. As in the MOF probe example, the total thickness of about 120 µm and wireless integration are critical. The following are being addressed: coatings such as Nafion, use of biopolymer protection, ensuring cytocompatibility (low immune response), minimizing phototoxicity, and tissue heating. The implantable MOF probe does cytotoxicity and immunofluorescence analyses. For in vivo or interstitial fluid, signal drift, background fluorescence or absorption, and interference from other molecules (ascorbic acid, uric acid, etc.) are major issues. Devices are increasingly evaluated in tissue or tissue-mimicking phantoms. The fiber SPR CuO sensor, for example, has low interference. The microneedle sensor works in phantom gel. Wireless data acquisition, continuous measurement during physiological or pharmacological events (e.g., deep brain stimulation), or on wearable patches collecting interstitial fluid or sweat are being demonstrated. The implantable probe with MOF monitors during stimulation in rats; wearable patches are being built for continuous sweat neurotransmitter (or analog) monitoring [192]. Eventually, solving these challenges depends on how well fluorescent nanomaterial-based sensing can integrate with emerging tools, as discussed in the following section.

7. Integration with Emerging Technologies

Fluorescent nanomaterial sensors for neurotransmitters have developed from fundamental photophysical designs to increasingly integrated analytical systems that utilize computational intelligence, hardware miniaturization, and physiologically relevant testing environments. The evolution of these probes is no longer limited to chemistry or optics alone; it now intersects with data science, electronics, and bioengineering [196]. Such convergence is essential to overcome long-standing challenges in neurochemical sensing, including overlapping spectral responses, transient signal dynamics, limited signal-to-noise ratios, and the need for in situ or in vivo applicability. Such systems depend primarily on three capabilities: (i) extracting chemically specific signatures from overlapping, time-dependent fluorescence signals, (ii) denoising and stabilizing low-photon recordings in real time, and (iii) validating algorithmic outputs against orthogonal chemical ground truth (Figure 6) [197].

Figure 6. Integrating artificial intelligence and machine learning towards efficient neurotransmitter sensing.

7.1. AI and Machine Learning for Fluorescence Signal Analysis and Pattern Recognition

Machine learning (ML) and deep learning (DL) provide the software backbone for turning complex fluorescence readouts from nanomaterial sensors into chemically meaningful time series and concentration estimates. Fluorescent nanoprobes often produce multidimensional outputs, full emission spectra that shift with environment and binding state, intensity transients with non-trivial kinetics, and spatially heterogeneous signals in cells and tissue. All of these are poorly handled by single-wavelength, linear calibration models. ML overcomes these limitations by learning non-linear mappings between high-dimensional fluorescence inputs and target outputs (identity, concentration, or event timing). This is being done by enabling: (i) spectral unmixing of overlapping emissions, (ii) separation of true chemical quenching/turn-on from optical mechanisms (IFE, scattering, autofluorescence), and (iii) event detection in low-photon, dynamically changing recordings. Data-driven unmixing networks (for example, UNMIX-ME and related DL unmixers) have demonstrated quantitative recovery of multiple fluorescent species from spectrally overlapping emissions and can be extended to arrays of nanomaterial probes that respond differentially to neurotransmitters. These methods are particularly powerful when combined with lifetime information or with training data that spans the probe’s environmental variability, since lifetimes are less sensitive to IFE and thus provide orthogonal contrast for ML models to utilize [198]. Kelich et al. combined systematic experimental screening of DNA sequences with ML-driven selection to discover ssDNA-SWCNT constructs that produced large, selective fluorescence responses to serotonin and enabled in vitro imaging of serotonin release [199]. This work shows that ML is not only a post-processing tool but can guide probe discovery and optimization, accelerating the identification of sequences or surface chemistries that maximize analyte-specific fluorescence modulation. Similarly, recent directed-evolution efforts for near-infrared serotonin nanosensors paired ML-guided candidate selection with high-throughput optical screening to improve sensitivity and selectivity of ssDNA-SWNT sensors for use in tissue. These probe-discovery examples demonstrate how ML shrinks the chemical search space and yields practical nanoprobes whose fluorescence dynamics are amenable to downstream algorithmic decoding. Beyond probe selection, ML improves live imaging readouts that track neurotransmitter release. Optical nanosensor “paints” and surface-anchored probes (for example, the AndromeDA dopamine nanosensor paint) produce localized, transient fluorescence responses when monoamines are released from vesicles; however, single-pixel intensity traces are noisy and easily confounded by motion, photobleaching, and background [200]. Deep denoising networks (e.g., DeepCAD-RT and related self-supervised models) and segmentation networks developed for live-cell fluorescence can markedly increase signal-to-noise ratio (SNR) and temporal precision, enabling detection of sub-second secretion events and more accurate spatial mapping of release sites. Combining denoising with supervised event detection yields pipelines that can recover phasic dopamine transients from movies that would otherwise be unusable, and these approaches have been validated in high-speed fluorescence imaging contexts to recover biologically meaningful kinetics [201]. Machine learning has also enabled multianalyte decoding from sensor arrays or multispectral probes. Pattern-recognition methods (LDA, PLSR) demonstrated for colorimetric nanoparticle arrays have direct analogues in fluorescence: when multiple nanomaterials with partially overlapping, analyte-dependent spectra are used in parallel, supervised models can learn the spectral fingerprint associated with each analyte and mixture composition. Recent nanozyme and MOF sensor-array studies used kinetic absorbance profiles combined with neural networks or PLS-based regressors to quantify neurotransmitters in serum and CSF, showing that dynamic, multivariate optical signatures can substitute for chemical specificity of single receptors. For fluorescence applications, these ideas translate into multiplexed panels where the full spectral-temporal trace forms the input to a regression or classification model that returns concentrations for several neurotransmitters simultaneously, a capability especially valuable where biochemical cross-reactivity is unavoidable [202].

7.2. Data-Driven Spectral Deconvolution in Multiplexed Detection

To monitor more than one neurochemical (or neurochemical signal) simultaneously with fluorescent nanomaterials imposes needs on spectral deconvolution. Fluorescent nanoprobes often employ multiple emitter types (e.g., QDs, CDs, upconversion nanoparticles) or use reference fluorophores in ratiometric schemes; each emits over a certain wavelength range, often overlapping. Adding in autofluorescence, scattering, and signal bleed-through, multiplexed detection becomes a deconvolution problem by separating mixed spectral signals into component channels that map to specific fluorophores and thus specific analyte interactions. Traditional linear unmixing methods assume known spectra, stable emission shapes, and clean background assumptions that often fail in biological settings with nanomaterials. Recent studies have employed data-driven or blind/semi-blind unmixing methods, often combined with machine learning or statistical modeling, to overcome these limitations. Smith et al. developed UNMIX-ME, a convolutional neural-network based framework capable of simultaneously unmixing multiple fluorescent species using both spectral and fluorescence-lifetime domains [198]. The model was trained on simulated hyperspectral fluorescence lifetime imaging datasets and validated experimentally with complex dye mixtures. It accurately recovered fluorophore abundance maps even when emission spectra overlapped by more than 80%, outperforming conventional least-squares spectral unmixing and lifetime-fitting approaches. This approach provides a direct computational template for multiplexed nanomaterial probes whose emissions partially coincide in wavelength but differ in lifetime or decay kinetics. Most of the studies employ dye fluorescence for demonstrating spectral deconvolution, which has a similar overlapping of emission bands to nanomaterial-based sensors. Multi-view ML approach by Wang et al. used both excitation and emission wavelengths by collecting emission spectra under different excitation conditions [203]. This was performed to produce multiple views of the same specimen. With this, they were able to more accurately distinguish highly overlapping fluorophores than would be possible from a single spectral measurement alone. They tested this on simulated data, but also on mixtures of fluorescently labeled Escherichia coli, showing that even for fluorophores with very similar spectra, both abundance and identity can be reliably recovered. This strategy could be especially useful when dealing with nanomaterial-based sensors whose emission shifts with environmental factors (pH, binding, quenching), because multiple excitation conditions can reveal differential response patterns. Sadashivaiah et al. introduced SUFI (Spectral Unmixing of Fluorescent Images), an automated pipeline that extracts spectral endmembers from multiplex fluorescence microscopy data and applies iterative non-negative least-squares unmixing [204]. Tested on mouse and post-mortem human cortical tissue containing up to 12 fluorophores, SUFI outperformed commercial algorithms by reducing bleed-through and autofluorescence artifacts. The data-driven extraction of endmember spectra without the need for prior references demonstrated a transferable solution for nanomaterial-based multiplex sensing in complex biological matrices such as brain slices or organoid models.

7.3. Wireless and Portable Readout Systems and Organ-on-Chip Integration

Translating fluorescent nanomaterial probes from benchtop instruments to practical neurochemical sensing requires an interface between the optical transducer and deployable hardware capable of real-time, wireless, or physiologically relevant operation. In parallel, the advent of organ-on-chip (OoC) systems offers an intermediate platform between isolated in vitro assays and complex in vivo experiments. Both directions of portable readout miniaturization and micro physiological integration aim to maintain the quantitative performance of fluorescence nanoprobes while meeting the constraints of biocompatibility, spatial confinement, and dynamic fluidic control. The foundation for wireless optical readout in neurochemical sensing was demonstrated by Kim et al., who engineered a miniaturized and wireless optical dopamine sensor based on QD-functionalized optical fibers [205]. The probe consisted of CdSe/ZnS QDs immobilized onto a silanized fiber tip, forming a dopamine-responsive photoluminescent interface through surface redox quenching. The sensor was integrated with a compact LED excitation and photodiode detection unit connected with a Bluetooth transceiver, enabling real-time, wireless optical transmission of dopamine concentration changes in vitro and ex vivo. Calibration experiments achieved a linear response in the 0.5–50 µM range with minimal interference from ascorbic and uric acids. This study established key design criteria such as stable surface immobilization, interference rejection, and optical-electronic coupling. Further miniaturization of optical platforms has enabled in vivo-compatible fluorescence detection. Zheng et al. developed a far-red genetically encoded dopamine sensor (HaloDA1.0) that enables real-time monitoring in zebrafish, mouse striatum, and brain slices using portable fiber-coupled photometry modules [206]. Utilizing the far-red spectral window (650 nm) effectively reduces tissue autofluorescence and scattering, allowing integration with low-power diode lasers and micro-photodetectors. While this system uses protein fluorophores rather than nanomaterials, advances in spectral engineering and hardware modularity provide useful design modules that can be integrated for nanomaterial-based platforms, particularly those using near-infrared QDs or SWCNTs as emissive elements. The study demonstrated sub-second response kinetics, ratiometric signal normalization, and simultaneous dual-color acquisition for multiplex neurotransmitter tracking, illustrating how portable fluorescence setups can achieve neural interface-level performance by means of fast receptor–ligand binding dynamics, synchronized dual-channel detection, and real-time signal processing.

On the microengineering side, organ-on-chip (OoC) systems have emerged as critical intermediates for validating fluorescent nanomaterial sensors under quasi-physiological conditions. Complementing this, Kieninger et al. reviewed strategies for embedding optical microsensors into organ-on-chip devices, highlighting the integration of waveguides, micro-LED excitation, and photodiode arrays for multiplex readout [207]. Their review demonstrated that fluorescence-based microsensors, when adapted to low-volume, high-flow environments, require specific design attention to excitation stability, photobleaching resistance, and temperature control. These results directly inform nanomaterial probe deployment in microfluidic circuits, particularly for sensors employing photostable nanomaterials such as nitrogen-doped carbon dots or upconversion nanoparticles.

7.4. Integration Challenges and Practical Bottlenecks

AI-assisted data processing and wearable or implantable fluorescence sensors are explored for neurotransmitter monitoring but their integration with nanomaterial-based fluorescent sensing platforms is currently technically challenging. One of the major bottlenecks is the variability of fluorescence signals in complex biological environments, in which photobleaching, signal drift, background autofluorescence, and inner filter effects complicate reliable feature extraction. AI algorithms are usually trained idealized or in vitro datasets mostly observed to struggle to generalize across fluctuating physiological conditions further limiting their advantages in real-time neurotransmitter sensing.

For wearable and implantable fluorescent sensors, material-related concerns further limit practical applications. Long-term photostability, biocompatibility, and immune response are the critical challenges, particularly for metal-containing nanomaterials and QDs. In addition, continuous excitation is required for fluorescence readout, which also increases risks of phototoxicity and thermal effects, thereby complicating chronic implantation. Successful integration of this technology, therefore, requires careful coupling of fluorescence mechanisms with device structure, ratiometric or lifetime-based sensing strategies. Hence, this will reduce dependence on absolute intensity fluctuations. Experiments combining fluorescent probes with microfluidic devices or optical fibers have been reported in the literature, but large-scale translation remains to be conducted, restricted by stability, calibration, and regulatory considerations.

8. Challenges and Limitations

Although remarkable advancements have been made in nanomaterial-based fluorescent sensing toward neurotransmitters, there are still certain theoretical and practical limitations that hamper the translational application of these sensors. The challenges mainly arise from the physicochemical properties of the sensing materials and underlying fluorescence modulation pathways, which are discussed in Section 2 and Section 3. Various classes of nanomaterials are explored for the sensing of neurotransmitters, exposing the absence of a universally optimal platform. Individual nanomaterials provide specific limitations, including scalability, long-term stability, and reliability in measurement repetition. Selectivity issues are among the prevalent challenges in complex biological matrices, irreversible signal outputs, and photodegradation.

(i) Long-term or high-intensity fluorescence monitoring is still restricted by photobleaching and optical stability, which are key technical challenges. Despite being initially bright, many carbon- and semiconductor-based probes suffer from photo-oxidation or ligand loss under prolonged excitation. This results in intensity fluctuation and the misinterpretation of the apparent concentration alterations. Approaches like surface passivation, silica, polymer shell encapsulation, and the anti-blinking core–shell QDs have been employed to alleviate this problem, albeit not completely successfully. In addition, in living systems, fluctuations of local pH values, ionic strengths, or temperatures can result in variations in quantum yield or emission wavelength and hamper the calibration. The photostability of the NIR emitters as well as of ratiometric reference channels is hence crucial for continuous long-term imaging or in vivo telemetry.

(ii) Molecular specificity is a parallel issue that is as challenging. Closely related catechol scaffolds within the same neurotransmitters, such as dopamine, norepinephrine, and epinephrine, are also common, as is their general redox potential and optical response overlap with other pathways, including serotonin or tryptophan derivatives. Therefore, if the mechanism of fluorescence quenching is solely electron transfer or π-π interaction, cross-reaction may occur. Real molecular selectivity can be achieved only by intelligent recognition layers, such as aptamers, MIPs, and enzyme cascades that transform subtle structural differences to discrete optical responses. However, even under these conditions, selectivity can become compromised in complex matrices containing both metabolites and oxidative intermediates. The development of such multilevel recognition systems discriminating at both the chemical and kinetic levels is an outstanding task.

(iii) In addition, complex biological backgrounds interfere heavily. Two of the limitations in the use of optical spectroscopy as a diagnostic technique for blood are that intrinsic chromophores are autofluorescent, turbid scattering tissues produce less distinguishable spectral baselines, and result in low signal-to-noise ratio measurements. Protein fouling in biofluids, such as serum or cerebrospinal fluid, and nanoprobe aggregation may result in emission quenching or nonspecific quenching. Such effects frequently obscure low-amplitude, but biologically relevant, neurotransmitter oscillations. While advanced antifouling coatings, the design use of NIR optical windows, and lifetime-based detection strategies can help to compensate for these effects, harmonization between instruments and biological preparations is still missing.

(iv) The toxicity and biocompatibility of nanomaterials are also a challenge. Cadmium and other heavy metal-based QD or some surface linkers may also cause oxidative stress, cell membrane disorder, and inflammatory stimuli. Even CDs or graphene derivatives might produce ROS at high excitation densities, and accumulate in lysosomes. The long-term destiny, removal, and possible disruption of the degradation products in neural tissues are not well known. Moving toward carbon nanostructures without dopants, degradable silicon-based emitters, or peptide-stabilized nanoclusters is an important step toward clinically compatible neuroprobes.

(v) Last of all, scalability and reproductivity in nanomaterial synthesis are still bottlenecks. A significant challenge with laboratory syntheses is the batch-dependent heterogeneity in size, surface functionalization, and photophysical properties, which impact sensitivity as well as calibration. Inter-laboratory reproducibility is also hampered by variations in precursor purity, reaction temperature regulation, and purification methods subsequent to synthesis. Scalable and standardized manufacturing routes (desirably solvent-free) to upscale production between laboratory, commercial, and regulatory stages with continuous-flow operations, including embedded quality control analytics, are required for translation of these sol–gel chemistries. Without this, even the most sensitive fluorescein probes will be limited to proof-of-principle experiments.

The aforementioned challenges, particularly stability, scalability, and clinical translational possibility, are the key bottlenecks that refrain the practical application of fluorescent nanosensors for neurotransmitter detection. These limitations arise from the properties and mechanisms of the same material that gives high sensitivity. The following future directions section proposes ideas for prioritizing the challenges and explains how the advances can move forward the laboratory-scale performance into clinical applications.

9. Future Perspectives

The combination of nanomaterial-based fluorescent sensing for neurotransmitters with advancements in real-time analysis, smart material development, and neurotechnological integration will likely characterize the next decade of device miniaturization. Coming advances will not only optimize nanoscale optical transduction, but also conceptualize new modalities for information extraction, signal processing, and information utilization at the level of living systems from optical quantification of fluorescence. One primary focus is the deployment of a multiplexed sensing platform that can monitor multiple neurotransmitters at the same time in complex environments. However, existing sensing platforms face limitations in spectral overlap, cross-reactions, kinetics, and maintaining calibration. Incorporating data-driven unmixing algorithms and adaptive calibration will turn these systems from single-target assays to full structured nanoscale chemical imaging platforms, thus taking optical microdialysis to the nanoscale. Also, as paradigm-changing as any of the topics just mentioned is the dream of real-time, low-invasiveness neurochemical recording. Advanced flexible optical fibers, micro-LED exciters, and near-infrared quantum emitters have recently enabled the integration of fluorescence sensing inside wireless or implantable interfaces. Sensors could then continuously operate in the interstitial environment of the brain without chronic immune activation by coupling these nanoprobes with transient or biodegradable materials. Ideally, this type of real-time feedback could finally facilitate changes at the spatial scale of dynamic neurotransmission in vivo beyond the temporal resolution of fast electrophysiology and into the slower domain of biochemical assays. Simultaneously, nanomaterial fluorescence technologies are going to provide us with access to luminous neurochemical diagnostics of a personalized nature. The potential for measurement of small changes in the levels of neurotransmitters, in readily available body fluids like serum, saliva, or cerebrospinal fluid, which would enable patient stratification of neurological and psychiatric disorders. Due to potential coupled with microfluidic handling in conjunction with a smartphone-based readout, portable fluorescence biosensors can be envisaged as personalized neurochemical signatures useful for later diagnosis, therapy monitoring, and drug-response prediction. Ultimately, the line between sensing and intervention could become so thin that fluorescent nanomaterial probes will eventually become elements of closed-loop neurotherapeutic systems. When optical sensors are combined with neural stimulation or drug-delivery modules, neurotransmitter imbalance can be addressed using feedback control. This notion parallels the nascent field of smart neuroprosthetics but requires a biochemical layer one that exploits the rapid, reversible, and quantitative nature of fluorescence in place of just electrical activity.

10. Conclusions

Fluorescence-based nanomaterial sensors have revolutionized the way of perceiving and quantifying neurochemical communication. Over the past decade, the convergence of nanoscience and neurobiology has yielded probes with extraordinary sensitivity, selectivity, and temporal resolution. Carbon-based, metallic, semiconducting, polymeric, and hybrid nanostructures have each demonstrated the capacity to transduce subtle neurotransmitter dynamics into measurable optical signals. Mechanistic pathways such as FRET, PET, IFE, and AIE have emerged as guiding principles for rational probe design, allowing fluorescence modulation to be tuned by molecular recognition events with nanometer precision. These observations reaffirm that understanding fluorescence mechanisms is not merely academic but foundational for building intelligent sensing systems. Future progress will hinge on integrating these mechanistic principles with advanced nanomaterial architectures capable of operating stably in physiologically complex environments. The next generation of probes must combine high quantum efficiency, biocompatibility, reversibility, and spatial and temporal accuracy to enable real-time monitoring of neurotransmitters within living neural circuits. The synergy between nanotechnology, AI, and microfluidic or optogenetic platforms will drive a new paradigm in neuroanalytical science. ML-assisted signal interpretation, implantable and wearable fluorescent interfaces, and multi-modal sensing will together transform fluorescence detection into a true window into brain chemistry. Ultimately, nanomaterial-enabled fluorescence sensing is not just advancing neuroscience, but it is humanizing it, lighting the invisible language of neurons through light, and providing opportunities to understand, diagnose, and modulate the neural code that governs human thought and emotion.