Near-Infrared Optical Constants and Guided-Mode Benchmarking of High-Index MoSe₂ for Nanophotonics

Abstract

The integration density of photonic integrated circuits is fundamentally limited by evanescent field overlap and subsequent inter-channel crosstalk. Layered transition metal dichalcogenides (TMDCs) bypass these confinement constraints through intrinsic optical birefringence and high refractive indices. Here, we report the near-infrared optical constants and waveguide dispersion of molybdenum diselenide (MoSe₂). Ellipsometry performed on centimeter-scale crystals yields an in-plane refractive index of 4.1–4.7 over 1000–2000 nm, with an extinction coefficient close to the sensitivity limit of the fit away from strong excitonic resonances. To validate the anisotropic dielectric tensor at the device scale, scattering-type scanning near-field optical microscopy (s-SNOM) was utilized to map the propagation of transverse-magnetic modes in 235 nm thick exfoliated flakes. Spatial Fourier analysis of the edge-scattered near-field interference yields effective mode indices that precisely match the modeled dispersion. Using the verified dielectric tensor, finite-element simulations demonstrate that single-mode MoSe₂ waveguides optically outperform equivalent tungsten disulfide (WS₂) benchmarks. The enhanced evanescent field suppression in the claddings of MoSe₂ waveguide increases the coupling length by a factor of 3.5, reducing the required routing pitch and enabling a 12.5% direct increase in on-chip integration density. The results identify MoSe₂ as a high-index anisotropic platform for compact waveguiding in the near-infrared.

1. Introduction

Dense photonic integrated circuits require tight optical confinement to minimize individual device footprints [1,2]. In conventional isotropic platforms, such as silicon and silicon nitride, physical scaling is bottlenecked by evanescent field leakage, which causes severe optical crosstalk at sub-wavelength routing pitches [3]. Layered van der Waals (vdW) materials with a strong in-plane covalent bonding and weak out-of-plane vdW forces offer a structural solution to this confinement limit [4]. VdW materials have become a central material class in modern nanophotonics because they combine strong light–matter interaction with crystallographically defined optical anisotropy and broad compatibility with heterogeneous integration [2,3,5,6,7,8]. Within this family, transition metal dichalcogenides (TMDCs) are especially relevant for guided-wave and resonant nanophotonics because bulk and multilayer crystals can simultaneously provide a high refractive index, giant uniaxial optical birefringence, and low loss over selected spectral windows [4,9,10,11,12,13,14]. From an integrated photonics perspective, these properties are not only of fundamental interest. The refractive-index contrast and the anisotropic dielectric response directly determine modal confinement, single-mode design limitations, evanescent field penetration, and the strength of coupling between adjacent optical channels [5,9,15,16]. As a result, reliable optical constants in the near-infrared around the telecommunication band are crucial for the realistic benchmarking of TMDC waveguides, couplers, and other building blocks of photonic integrated circuits.

Molybdenum diselenide (MoSe₂) is a particularly interesting material of the high-index TMDC class [5]. Prior studies have established that MoSe₂ can sustain guided exciton–polariton transport in crystal flake waveguides closer to the visible and excitonic spectral range, and highly confined optical modes and high nonlinear response in epitaxially grown MoSe₂ in the near-infrared range [12,17,18]. The combination of a high in-plane refractive index (n_ab > 4), low absorption, and giant optical anisotropy, birefringence Δn = n_ab − n_c ~ 1, in crystal bulk MoSe₂ enables the localization of the electromagnetic field at subwavelength scales and provides a single-mode operation in compact photonic waveguides [4,5,9]. In this regard, the design of low-loss near-infrared photonic components requires a device-oriented description that links anisotropic optical constants, experimentally observed guided modes, and engineering figures of merit such as coupling length. However, determination of the dielectric permittivity of vdW crystals is non-trivial, since the studied samples—microcrystals, i.e., flakes—obtained by mechanical exfoliation from the original crystal, often have dimensions of up to several tens of micrometers [19,20]. This restricts the utilization of traditional methods such as spectroscopic ellipsometry and reflectometry for determining the optical constants over a wide spectral range. Moreover, these far-field optical methods have low sensitivity in determining the out-of-plane component of refractive index (n_c) [10]. In contrast, scattering-type scanning near-field optical microscopy (s-SNOM) enables local measurement of the optical response, excitation and propagation of guided modes in vdW materials with spatial resolution exceeding the diffraction limit, and allows verification of the numerical values of the anisotropy of optical constants obtained independently, using spectroscopic ellipsometry [9,10,16,20,21,22].

In this work, we present a detailed characterization of the optical constants and waveguide properties of the high-index crystal MoSe₂ in the near-infrared by combining the macroscopic spectroscopic ellipsometry, s-SNOM, and electromagnetic modeling to build and test an anisotropic optical model demonstrating the applications of MoSe₂ in nanophotonics. First, we extract the in-plane optical constants of a macroscopic crystal in the 1000–2000 nm range. Next, we visualize guided TM modes in an exfoliated MoSe₂ flake and compare the measured modal dispersion with transfer-matrix calculations. Finally, we use the experimentally supported model to benchmark ridge MoSe₂ waveguides against WS₂ counterparts in the single-mode regime.

2. Materials and Methods

2.1. Material Preparation, Morphological and Elemental Characterization

A commercially available ultra-flat large-area bulk MoSe₂ crystal grown by chemical vapor transport (CVT, 2D Semiconductors, Inc., Phoenix, AZ, USA) was used for ellipsometric characterization. The crystal composition was examined by energy-dispersive X-ray spectroscopy (EDS, Bruker QUANTAX EDX, Bruker Corporation, Billerica, MA, USA) in a scanning electron microscope setup (SEM, JEOL JSM-7001F, JEOL Ltd., Tokyo, Japan) working in secondary electron imaging mode. The value of the acceleration voltage was 15 keV. The EDS spectrum was analyzed using the software provided by Bruker. The crystal phase of MoSe₂ was assessed by the Raman spectrometer Horiba LabRAM HR Evolution (HORIBA Ltd., Kyoto, Japan) equipped with a confocal microscope with objective ×100/N.A. = 0.90 and 1800 lines/mm diffraction grating, working at an excitation wavelength of 532 nm. For near-field measurements, MoSe₂ flakes were obtained by mechanical exfoliation from the same parent crystal and transferred onto a SiO₂(285 nm)/Si substrate. Optical microscopy was used to identify flakes of sufficient lateral size, and atomic force microscopy (AFM, NT-MDT Spectrum Instruments, NT-MDT Ltd., Moscow, Russian Federation) in contact mode was used to determine the flake thickness. The cantilevers employed were NSA01 tips (TipsNano OÜ, Tallinn, Estonia), characterized by a spring constant of 5.1 N/m, a tip radius of less than 10 nm, and a resonant frequency of 150 kHz. Analysis of AFM scans was performed using Gwyddion (ver. 2.71) software.

2.2. Spectroscopic Ellipsometry

The optical constants of the macroscopic MoSe₂ crystal were measured with a variable-angle spectroscopic ellipsometer (V-VASE, J.A. Woollam Co., Linkoln, NE, USA) in the wavelength range 1000–2000 nm. A large-area crystal (~1 cm²) was required because the ellipsometer spot size was approximately ~1.5 mm. The ellipsometric parameters Ψ and Δ were recorded in reflection geometry at incidence angles of 50°, 55°, and 60° with a wavelength step of 5 nm. The data were fitted in WVASE using a parallel-sided slab model that included backside reflection. The dielectric response was parameterized with the Tauc–Lorentz dispersion model, which preserves the Kramers–Kronig consistency and has been broadly used for TMDC optical constants extraction [5,12]. The out-of-plane component of the refractive index n_c was determined using the imaging ellipsometer (Accurion EP4) in the nulling mode. Ellipsometry spectra were recorded for MoSe₂ flake with thickness of 100 nm and in the NIR spectral range. For ellipsometry analysis, the fitting method similar to the procedure described in previous works was used [10].

2.3. Scattering-Type Scanning Near-Field Optical Microscopy

Near-field imaging of guided modes was performed with a commercial s-SNOM system (neaSNOM, Neaspec GmbH, Haar, Germany) operated in reflection mode. A tunable continuous-wave fiber-coupled laser (Agilent 81600B) with a range of 1460–1640 nm was used as the excitation source. The optical probe was a metalized AFM tip (NanoWorld, ARROW-NCPt-50) oscillating at Ω ≈ 280 kHz. A pseudo-heterodyne interferometric scheme was employed, and the detector signal was demodulated third harmonic to isolate the background-suppressed near-field amplitude and phase [23]. The excitation polarization was chosen to launch TM-polarized guided modes in the flake. To select a suitable planar waveguide thickness, the TM-mode dispersion of the MoSe₂/SiO₂/Si stack was first evaluated by the transfer-matrix method [24] for the measured in-plane response and the adopted out-of-plane refractive index.

The near-field fringe pattern was analyzed in the Fourier domain using line profiles extracted perpendicularly to the flake edge. In accordance with the established interpretation of guided-mode s-SNOM in anisotropic TMDCs flakes, the observed pattern is attributed to interference between the incident field and a tip-launched guided mode that is scattered at the flake boundary. The Fourier peak position yields the apparent normalized in-plane momentum n_s-SNOM. The effective mode index was then extracted as $n_{\mathrm{eff}} = n_{\mathrm{s}-\mathrm{SNOM}} +\cos \alpha ·\sin \beta$, where α is the angle between the illumination wavevector and its projection onto the sample plane, and β is the angle between the in-plane projection and the flake edge. For the present geometry, α = 45° and β = 90°.

2.4. Numerical Modeling

Ridge waveguides with MoSe₂, WS₂ and Si cores on a SiO₂ substrate and air cladding were analyzed using the finite-element method in COMSOL Multiphysics. The simulation domain was 8 × 8 μm with PEC boundary. Mesh size varied from 20 nm at the waveguide core to 250 nm near the domain boundary. The crystal optic axis was assumed to be normal to the substrate, consistent with the layered-flake geometry. To quantify coupling between neighboring waveguides, even and odd supermodes of a coupled-waveguide pair were calculated for different center-to-center spacings d. The coupling length was then obtained from [25]:

$$L_c = \lambda /\left[2\left|n_{\mathrm{even}}- n_{\mathrm{odd}}\right|\right].$$

Coupling length was optimized at core widths and heights, at which waveguide operates in the single-mode regime. Since L_c grows with the increase in core height, the optimal waveguide size is found at the line separating single and double-mode operation regimes.

3. Results and Discussion

For the initial determination of the in-plane optical constants, spectroscopic ellipsometry was applied to the molybdenum diselenide (MoSe₂) crystal. Measurements were performed using a Woollam V-VASE ellipsometer in the wavelength range λ = 1000–2000 nm, which is important for telecommunications applications. A large-area crystal (~1 cm²), synthesized by CVT, was chosen as the MoSe₂ sample. In the experiment, the choice of a macroscopic crystal was dictated by the minimum spot size of the ellipsometer measurement beam, which is approximately ~1.5 mm in diameter. Measurements of the ellipsometric parameters Ψ and Δ were performed in reflection mode (see Methods). To determine the in-plane refractive index n_ab and extinction coefficient k_ab of the MoSe₂ crystal, a layered model including backside reflection was used. For the dielectric function, the Tauc–Lorentz oscillator dispersion model was employed in fitting the Ψ and Δ spectra, providing analytical satisfaction of the Kramers–Kronig relations, similarly to approaches used for other TMDCs, using the WVASE software [9,10]. The results of the ellipsometry data analysis for MoSe₂ are presented in Figure 1a. The in-plane refractive index n_ab varies smoothly in the range of 4.1–4.7, remaining well above the values typical of conventional dielectric photonic materials in the near-infrared [5,10]. At the same time, the fitted extinction coefficient k_ab in the near-infrared window is below the sensitivity of the present ellipsometric fit, which is consistent with previous studies [12,18]. The performed ellipsometry method with a large-area MoSe₂ crystal is only effective for determining in-plane optical constants, but not the out-of-plane component n_c, due to the high uncertainty in the sample thickness. Thus, to determine the out-of-plane refractive index n_c with sufficient accuracy to take into account the anisotropy in further MoSe₂ waveguides studies, the ellipsometry of the separate flake with a certain thickness was employed (see Methods), yielding the results in Figure 1a within ~0.1 agree with n_c curve from reference [12].

The s-SNOM technique allows for the independent verification of the optical constants of vdW crystals and confirmation of the magnitude of optical anisotropy; s-SNOM was used to measure the propagation of photonic modes in MoSe₂ [10]. To this end, planar waveguides were fabricated by mechanically exfoliating MoSe₂ flakes onto a SiO₂(285 nm)/Si substrate from a macroscopic crystal. Suitable MoSe₂ flakes with sufficient area and the required thickness for the propagation of the fundamental photonic TM mode in the wavelength range near 1500 nm were identified using optical microscopy and AFM. The range of thicknesses required for the propagation of a single TM mode was determined by numerical calculation using the transfer-matrix method [24] and the optical constants of MoSe₂ in Figure 1a, yielding thickness values from 200 to 370 nm. This choice is important because it enables a direct comparison between measured and calculated modal dispersion without the ambiguity introduced by higher-order guided states. The AFM step profile on a chosen flake yielded a thickness of 235 ± 5 nm (see inset in Figure 1c), which falls inside the calculated single-mode interval for the planar TM mode near λ = 1.5 μm and was selected for s-SNOM measurements. The roughness of 0.29 nm (Figure 1e) confirms an atomically flat top surface which would result in reduced defect-related losses in waveguiding applications. In addition, the EDS spectrum in Figure 1b confirms the expected stoichiometry of the parent crystal, with measured atomic fractions of 34.7% for Mo and 65.3% for Se, corresponding closely to the 1:2 composition of MoSe₂. To confirm the crystal phase, we applied Raman spectroscopy measurements of MoSe₂ flake shown in Figure 1c, which correspond to bulk 2H-MoSe₂ and are characterized by two vibrational modes: the in-plane E_1g at ~168 cm⁻¹ and the out-of-plane A_1g peak at ~242 cm⁻¹ [26]. Bulk 2H-MoSe₂ has an indirect band gap of ~1.2 eV [5,12]. The photon energies (0.76–0.84 eV) considered in the work for studying the propagation of photonic modes in 2H-MoSe₂ are below the fundamental band gap, which supports the use of a low-absorption NIR dielectric window.

Next, to study photonic modes in MoSe₂ planar waveguides, the s-SNOM metallic tip is illuminated with laser radiation in the 1460–1640 nm range in TM polarization (see Section 2.3 for more details). Scanning the tip perpendicular to the crystal edge allows mapping of interference fringes. The resulting s-SNOM amplitude and phase images recorded near the flake edge at wavelengths of 1475 and 1625 nm are shown in Figure 2a. Both wavelengths reveal a clear sequence of near-field fringes extending inward from the flake boundary, which is consistent with the excitation of a guided TM₀ mode in the planar MoSe₂ waveguide. The fringe periodicity is directly related to the mode wavelength (λ_p) and, consequently, to the effective mode index (n_eff = λ₀/λ_p, where λ₀ is the free-space wavelength). The effective refractive index of the propagating TM mode in an anisotropic planar MoSe₂ waveguide depends on both the in-plane (n_ab) and out-of-plane (n_c) refractive indices. Thus, by determining n_eff and comparing it with theoretical calculations of the dispersion curve using the transfer-matrix method, it is possible to verify the anisotropic optical constants. To determine n_eff, complex fast Fourier transform of line profiles extracted from the near-field images was performed and the corresponding Fourier spectra are shown in Figure 2b. The dominant peaks (at q(k₀) ~ 1.1) assigned to the TM₀ mode were used to determine the in-plane modal momenta, while additional peaks in the Fourier spectra arise due to the propagation of the wave in the air and do not affect the position of the peak corresponding to the desired TM₀ mode. After applying the geometric frequency shift described in Section 2.3, the extracted effective indices at 1475 and 1625 nm were compared with transfer-matrix calculations of the energy (E ~ 1/λ₀)-momentum (q = 1/λ_p) dispersion relation of the waveguide mode for the anisotropic MoSe₂/SiO₂/Si stack. As shown in Figure 2c, the experimental effective indices’ points follow the calculated dispersion curve. This agreement does not by itself constitute a full independent reconstruction of the complete dielectric tensor, because the out-of-plane response was measured from an exfoliated MoSe₂ flake, but it does provide an important consistency check: the measured in-plane optical constants n_ab, k_ab, n_c, and the flake thickness together describe the observed guided-mode dispersion without the need for further adjustable parameters. Considering that the uncertainty in the determination of the n_eff of guided modes is about 0.05 (Figure 2c), we find that the agreement between s-SNOM measurement results and transfer-matrix calculations means that the obtained n_c values’ error is below 0.08 for wavelengths of 1475 and 1625 nm. Therefore, both the literature out-of-plane index n_c values from reference [12] and our new values directly describe the observed guided-mode dispersion in the telecom band without the need for further adjustable parameters.

Using the experimentally supported optical model, we next evaluated the prospects of using MoSe₂ in nanophotonics by numerical modeling of the behavior of waveguides based on this material. Characteristics of MoSe₂ waveguides were compared with those of WS₂—another TMDC with a high refractive index and optical anisotropy—whose waveguiding performance has previously been shown to be better than silicon, the backbone material of modern nanophotonics [9]. First, the effective indices of the supported waveguide modes were calculated (see Methods). This allowed determination of the number of supported waveguide modes as a function of the waveguide core width (w) and height (h), as well as the construction of effective-index maps for the fundamental mode, i.e., the mode with the highest index. From the nanophotonic viewpoint, the effective index is an important parameter characterizing the degree of field confinement within the waveguide core. In particular, it determines the size of resonators and modulators, and also governs radiation losses in bent waveguide sections. Furthermore, the higher the effective index, the faster the evanescent field decays in the waveguide cladding, leading to reduced crosstalk between adjacent waveguides and, consequently, enabling denser integration of waveguides on a chip, thereby enhancing the performance of photonic integrated circuits [9]. In Figure 3, the comparison of the effective-index maps for MoSe₂ and WS₂ rectangular waveguides shows that the MoSe₂ core supports modes with a higher effective index at smaller core dimensions, which is expected due to the higher refractive index of MoSe₂. On the other hand, for equal dimensions, MoSe₂ waveguides support more modes, whereas in practice, a single-mode regime is important, as it allows better control of signal transmission through the waveguide. Nevertheless, in the single-mode regime, i.e., at smaller core dimensions, MoSe₂ waveguides exhibit a higher effective index than WS₂ waveguides, suggesting weaker crosstalk compared to WS₂.

To test this hypothesis, the coupling length L_c, which characterizes the mutual influence of adjacent waveguides, was calculated. After propagating this distance along the waveguide, the mode completely transfers from one waveguide to the neighboring one (see Methods for the details on crosstalk calculations). The calculation results show that for a fixed waveguide core width w, L_c increases with core height; therefore, the optimal core dimensions should lie on the cutoff curve of the second waveguide mode (the white curve in Figure 3). For a fixed center-to-center distance d = 1 μm between waveguides, the maximum achievable coupling length L_c for MoSe₂ waveguides exceeds 15 mm, which is approximately 3.5 times higher than the same value for WS₂ waveguides, which is 4.3 mm. To provide more context, the maximum L_c for similar silicon waveguides is just 0.59 mm, which is almost an order of magnitude below the same value for WS₂.

A more detailed investigation of the dependence of L_c on d (Figure 4) shows that the advantage of MoSe₂ over WS₂ and over Si increases as the distance between waveguides grows. At the same time, if we compare the minimum distance between waveguides at which L_c = 10 mm, which is sufficient for constructing global interconnects on a chip, then MoSe₂ (d_min = 960 nm) outperforms WS₂ (d_min = 1080 nm) by 12.5% and silicon by (d_min = 1350 nm) by 40%. This enhancement is very significant and hard to achieve considering the near-exponential dependence of the coupling length on the distance between waveguides, which is caused by the exponential decay of the waveguide mode field outside the waveguide core. Moreover, there exist methods for further increasing integration density by engineering the space between waveguides, which can not only reduce the distance between waveguides but also almost completely suppress waveguide coupling [25,27].

Looking forward, the availability of scalable, high-index MoSe₂ structures provides a material platform for realizing all vdW microcavity photonics, high-Q resonant photonics and the integration of heterogeneous photonic circuits, etc. [28,29]. Such scalability has remained hard to achieve until now, as previous demonstrations of TMDC-based structures have predominantly relied on exfoliated flakes, whose lateral dimensions are limited to tens or hundreds of micrometers. For practical mm-cm photonic routing, MoSe₂ waveguides would require larger-area crystals or wafer-scale grown TMDC films with controlled thickness and roughness. However, recent advances in inch-scale MoSe₂-layer growth via molecular beam epitaxy (MBE) and subsequent patterning using electron lithography ensure a level of scalability that is crucial for the industrial fabrication of nanophotonic structures [18]. Given that grown 3D MoSe₂ layers exhibit optical properties comparable to those of exfoliated crystals, our results, aimed at clarifying the anisotropic optical properties, assessing the integration density of MoSe₂ based waveguides in comparison with other TMDCs and providing the design basis for such devices, establish MoSe₂ as a promising platform for integrated nanophotonics in the telecommunication range.

4. Conclusions

We have evaluated bulk-like MoSe₂ as a near-infrared waveguiding material by combining spectroscopic ellipsometry, s-SNOM near-field imaging, and electromagnetic modeling. The fitted in-plane refractive index remains between 4.1 and 4.7 from 1000 to 2000 nm, while the fitted extinction coefficient is close to zero within the sensitivity of the fit. Guided TM₀ modes imaged at 1475 and 1625 nm yield effective indices consistent with transfer-matrix calculations based on the measured in-plane response, the measured flake thickness, and an out-of-plane index. This agreement supports the internal consistency of the anisotropic optical model. Waveguide benchmarking further showed that, in the single-mode regime, MoSe₂ provides higher effective indices and longer coupling lengths than another high-refractive-index anisotropic material, WS₂. This leads to a reduction in crosstalk by approximately a factor of 3.5, demonstrating that MoSe₂ waveguides are more compact and effective, which can be used to increase the integration density and reliability of photonic circuits. These results support MoSe₂ as a high-index anisotropic material platform for compact near-infrared photonic routing.