Skip to Content
PhotonicsPhotonics
  • Article
  • Open Access

28 August 2026

15 Pages

Vortex Beam-Coupled Cassegrain Front-End Free-Space Optical Links with LSTM and Transformer-Based Signal Recovery

,
and
Department of Information and Telecommunication Engineering, Incheon National University, Incheon 22012, Republic of Korea
*
Author to whom correspondence should be addressed.

Abstract

Free-space optical (FSO) communication provides high-capacity wireless transmission but suffers from reduced optical coupling efficiency when compact Cassegrain telescopes are employed because the secondary mirror blocks the central portion of the incident beam. This paper proposes a vortex beam-assisted FSO communication system that combines aperture-matched optical coupling with machine learning-based signal recovery. Unlike a conventional Gaussian beam, the annular intensity distribution of a Laguerre–Gaussian vortex beam is matched to the unobstructed annular aperture of a centrally obscured Cassegrain telescope, thereby reducing obstruction-induced optical loss. The coupling characteristics are analyzed using an annular aperture overlap model and experimentally validated in a 100 m free-space optical link employing Cassegrain transmitter and receiver front-ends. The mean measured telescope-output power is increased by more than 30% over the Gaussian reference. To overcome the system-induced signal aliasing, Transformer and long short-term memory (LSTM) equalizers are optimized and applied. Both models substantially outperform optimized threshold detection, while the LSTM achieves the lowest observed error rate with markedly fewer multiply–accumulate operations than the Transformer. These results show that aperture-matched optical coupling and computationally efficient sequence equalization, such as LSTM is a crucial component of compact, high-performance telescope-assisted FSO systems.

1. Introduction

Compact long-range free-space optical (FSO) terminals must satisfy two requirements at the same time: they must deliver sufficient optical power through a small optical package and recover data reliably despite the bandwidth and alignment limitations of practical hardware. FSO offers high-capacity, license-free, and spatially confined links, making it attractive for terrestrial and satellite communication [1,2,3]. Cassegrain telescopes are well suited to compact terminals because their folded optical paths combine a large aperture with a long effective focal length. Their secondary mirrors, however, create a central obstruction that blocks the highest-intensity portion of a conventional Gaussian beam. This mismatch reduces the useful power transmitted through the telescope precisely when compact terminals have little link margin to spare.
Vortex beams and their generation have been widely studied in orbital-angular-momentum optics [4,5,6], and this line of research extends to double vortex beams carrying two phase singularities. Guo et al. resolved the magnitudes, signs and spatial distribution of the two topological charges by the tilted-lens method and showed that the average orbital angular momentum decreases nonlinearly with the singularity separation although the total topological charge is unchanged [7]. Chen et al. extended the treatment to partially coherent sources in oceanic turbulence [8] and then to radially polarized fields [9]. These works establish the singularity configuration, coherence and polarization as means of shaping the transverse intensity distribution, which is the quantity governing the annular-pupil overlap of Section 2, although the resulting fields are not circularly symmetric about the beam axis and are not coupled through an obscured pupil. Data-driven receivers have also been investigated for impaired OAM links [10]. In the present system, however, the vortex beam is not used for spatial multiplexing or preservation of an ideal OAM eigenmode; it is used as aperture-matched illumination. This distinction is important because increased telescope-output power alone does not remove the temporal distortion and interference introduced by the bandwidth-limited modulation and detection chain. A compact terminal therefore also requires an efficient sequence equalizer.
Previously, pin-like structured beams reduce clipping at small, unobscured circular receivers by reinforcing a narrow on-axis lobe [11,12], which is the opposite of the intensity distribution required when the optical axis is blocked. Cassegrain studies have analyzed obscuration, hollow-beam propagation, fiber coupling, coherent reception, and transmission-efficiency improvement [13,14,15,16,17], but have not jointly demonstrated pupil-matched annular illumination and low-complexity intensity-modulation/direct-detection (IM/DD) equalization. Reference [18] proposed matching the dark core of a vortex source to the secondary-mirror obscuration of a Cassegrain receiver by optimizing the topological charge, and verified the gain in power efficiency on a 6 mm annular aperture over a 0.75 m path under closed-loop adaptive optics [18]. This approach was later extended to a two-Cassegrain-telescope configuration and evaluated numerically under the H-V 5/7 turbulence model [19]. Both establish the optical principle used here but assess it through power coupling efficiency alone; no modulated data are transmitted, so whether the gain in collected power reduces the bit error rate of an intersymbol interference-limited link is not addressed. Annular vortex distributions have also been exploited for mode separation and direct detection at the detector or mode-analysis plane [20,21,22], rather than to reduce loss at the telescope pupil.
Sequence models have also been applied to the temporal side of optical wireless links. Reference [23] combined an LSTM with an attention decoder to detect hybrid modulated faster-than-Nyquist signals over a turbulence channel, reporting a 1 dB gain over a back-propagation network and 2.5 dB over maximum-likelihood sequence estimation. Recurrent layers have also been used away from the detection path, for instance, to predict user mobility and receiver orientation in order to optimize the beamforming and bias currents of an SLIPT-enabled indoor visible-light system [24]. The present work differs in three respects. The intersymbol interference addressed here is not introduced deliberately for spectral efficiency but arises from the transmitter modulation bandwidth, and it is measured on a physical 100 m link rather than generated from a channel model. The equalizer is compared against a Transformer using multiply–accumulate counts derived from the architecture, which are independent of the execution platform.
This work addresses the spatial and temporal impairments as complementary parts of one compact FSO link. The annular-pupil overlap is modeled using established analyses of obscured apertures and telescope coupling [25], and the beam profiles are related to Gaussian and Laguerre–Gaussian descriptions [26,27,28]. A vortex phase plate is then evaluated in a 100 m IM/DD link with Cassegrain transmitter and receiver front-ends. The optical stage is assessed through beam profiles and measured telescope-output power, while the residual bandwidth-induced waveform distortion is recovered using optimized long short-term memory (LSTM) and Transformer equalizers. Thus, the central question is not whether structured light or machine learning is beneficial independently, but whether an aperture-aware optical front end and a computationally efficient temporal equalizer can jointly improve a compact telescope-assisted link.
The main contributions are threefold. First, the annular matching principle established for Cassegrain receivers is recast as a design criterion specifying an admissible range of ring radii, with an explicit margin for alignment error and beam wander, rather than a single efficiency-maximizing optimum. Second, Gaussian and vortex illumination are compared experimentally in an operating IM/DD link over a 100 m path with Cassegrain transmitter and receiver front-ends so that the coupling gain is measured through its effect on the received waveform rather than as optical power efficiency alone. Third, LSTM and Transformer equalizers are optimized and compared in terms of bit error rate and multiply–accumulate complexity, establishing that LSTM in this case outperforms Transformer in both communication performance and complexity. Table 1 condenses the closest research directions and highlights the unresolved gap addressed by the proposed optical-digital design.
Table 1. Positioning of the proposed system relative to the closest optical and communication approaches.
The remainder of this paper is organized as follows: Section 2 develops the aperture-overlap model and matching condition. Section 3 describes the 100 m experiment, dataset construction, equalizers, and complexity metrics. Section 4 presents the optical-coupling and signal-recovery results, and Section 5 concludes the paper.

2. Aperture-Matched Vortex Beam Coupling

Cassegrain telescopes use a concave primary mirror and a convex secondary mirror to obtain a long effective focal length in a compact folded optical path [13,17]. The large clear aperture is useful for FSO transmission and collection, while the folded geometry reduces terminal length. The same secondary mirror produces a central obscuration and converts the pupil from a disk into an annulus. Light incident on the obstructed region is lost, and the obscuration also modifies the diffraction pattern. The optical front end should therefore be designed for the usable annular pupil rather than for an unobstructed circular aperture. The transmitter (Celestron NexStar 8SE) and the receiver (Celestron 127EQ3M-MAK5) use Cassegrain telescopes. Figure 1 summarizes their folded optical paths. In the Gaussian reference, power is concentrated near the optical axis. The vortex phase plate instead produces an annular beam with reduced on-axis intensity. The telescope aperture can be approximated as an annular aperture [29,30]. The corresponding pupil function is as below:
P ( r ) = 1 ,     r o b s < r < R 0 ,     o t h e r w i s e
where r denotes the radial coordinate on the entrance pupil, R is the radius of the clear aperture of the primary mirror, and r o b s represents the radius of the central obstruction introduced by the secondary mirror. When an incident optical beam with transverse intensity distribution I ( x , y ) illuminates the telescope, only the optical power transmitted through the annular pupil contributes to the received signal. Accordingly, the optical coupling efficiency of the telescope is defined as [27]
η t e l = ∬ A I x , y   d x d y ∬ R 2 I x , y   d x d y
= ∫ r o b s R I r   2 π r   d r ∫ 0 ∞ I r   2 π r   d r .
Figure 1. Folded optical configuration of the Cassegrain telescope and conceptual comparison of the incident Gaussian and annular vortex fields at the telescope input. The Gaussian beam concentrates power on the axis, where it overlaps the central obstruction, whereas the vortex beam redistributes power toward the transmissive annular pupil.
Equation (3) shows that the central obstruction reduces the collected optical power and the optical coupling efficiency for beams whose intensity is concentrated near the optical axis. A fundamental Gaussian beam has the following radial intensity distribution:
I G r = I 0 exp − 2 r 2 w 2 ,
where w denotes the beam waist. Since the maximum intensity occurs at the beam center ( r = 0 ), the Gaussian beam overlaps significantly with the secondary-mirror obstruction, resulting in reduced optical coupling efficiency.
In contrast, a Laguerre–Gaussian vortex beam with azimuthal index l exhibits a doughnut-shaped intensity profile given by [5,27]
I l r = I 0 ( 2 r 2 w 2 ) l exp − 2 r 2 w 2 .
The radius corresponding to the maximum intensity is obtained by solving d I l ( r ) d r = 0 , which yields [28]
r p e a k = w l 2 .
Equation (6) indicates that the ring radius increases with both the beam waist and the magnitude of the vortex topological charge l .
To maximize the optical coupling efficiency of the Cassegrain telescope, the intensity ring of the vortex beam should coincide with the transmissive annular pupil while avoiding the central obstruction. The basic matching condition is, therefore,
r o b s < r p e a k < R .
Considering finite ring thickness, beam wander, and optical alignment tolerances, a practical design criterion is [15]
r o b s + ∆ r < r p e a k < R − ∆ r
where ∆ r denotes the design margin.
Since r p e a k = w l / 2 , the topological charge determines the radial position of the intensity maximum and must be selected together with the transmitted beam size rather than independently. A low charge places the ring within the obscured central region, where it is blocked, whereas a high charge expands it beyond the clear aperture, where it is truncated.
Substituting Equation (5) into Equation (3) and setting u = 2 r 2 / w 2 reduces the overlap integral to a closed form,
η t e l l , w = γ l + 1 ,   2 R 2 w 2 − γ ( l + 1 , 2 r o b s 2 w 2 ) Γ ( l + 1 )
where γ and Γ denote the lower incomplete and complete gamma functions. The coupling efficiency is therefore not monotonic in l , and the design problem is the maximization of η t e l over the beam size w at the pupil plane and the topological charge for a given aperture geometry. Setting ∂ η t e l ∂ w = 0 gives the stationarity condition
( 2 R 2 w 2 ) l + 1 e − 2 R 2 / w 2 = 2 r o b s 2 w 2 l + 1 e − 2 r o b s 2 / w 2 ,
which states that at the optimum the marginal power lost by truncation at the outer aperture equals the marginal power lost to the central obscuration. Taking logarithms and introducing the obscuration ratio ε = r o b s / R yields
w o p t = R 1 − ε 2 l + 1 I n ( 1 ε ) ,     r p e a k o p t = R l ( 1 − ε 2 ) l + 1 I n ( 1 ε ) .
The optimal ring radius is thus determined by the obscuration ratio and the topological charge alone, and Equation (8) is recovered as the admissible region surrounding this optimum, with the margin ∆ r accommodating alignment error and beam wander. When the beam size at the pupil is fixed by the collimation optics, as in the present system, the remaining design variable is the topological charge. This is consistent with the charge optimization reported for Cassegrain receivers in [18,19]. For the telescope geometry used in this work (Section 3), l = 2 satisfies Equation (8) at both telescopes.
The matching condition of Equation (8) applies at the pupil plane of the receiving telescope, so for a link of length L the ring radius must be evaluated after propagation. The Gaussian envelope parameter of Equation (5) follows the standard propagation law
w L = w 0 1 + ( λ L π w 0 2 ) 2 ,     r p e a k L = w L l 2 ,
so that the distance-dependent design criterion becomes
r o b s + ∆ r < r p e a k ( L ) < R − ∆ r
The ring radius therefore remains a fixed multiple of w L throughout propagation, and aperture matching is preserved while L ≪ z R , with z R = π w 0 2 λ , degrading only once the expanded ring overfills the clear aperture. A larger topological charge increases the ring radius at every distance and consequently tightens the upper bound of the criterion, so w 0 and l must be chosen jointly with the intended link length. The margin ∆ r must additionally accommodate turbulence-induced beam wander, whose variance grows with propagation distance.
This condition can be satisfied by appropriately adjusting the beam waist w , the vortex topological charge l , and the telescope magnification.
Consequently, a properly designed vortex beam reduces the optical power blocked by the secondary mirror and improves the optical coupling efficiency compared with a conventional Gaussian beam. For quantitative comparison, the coupling improvement is defined as
G = 10 l o g 10 η t e l l η t e l 0 d B .
A positive value of G indicates that the vortex beam achieves higher coupling efficiency than the Gaussian beam. The received optical power of the FSO link is modeled as [1,2]
P r x = P 0 η t e l η p r o p L η r x η d e t ,
where P 0 denotes the transmitted optical power, η p r o p L is the free-space propagation efficiency over a transmission distance L , η r x is the receiver optical efficiency, and η d e t denotes the photodetector efficiency. The corresponding electrical SNR is expressed as [2]
S N R = ( R P D P r x ) 2 2 q e R P D P r x B + i t h 2 B ,
where R P D is the photodetector responsivity, q e is the electron charge, B is the receiver bandwidth, and i t h denotes the thermal noise current density. Equations (15) and (16) directly relate the optical coupling efficiency of the telescope to the received optical power and the communication performance of the FSO link.
Based on the above analysis, we hypothesize that a vortex beam whose ring radius satisfies the annular matching condition of Equation (8) achieves higher optical coupling efficiency than a conventional Gaussian beam in a centrally obscured Cassegrain telescope. The resulting increase in the received optical power is expected to improve the electrical SNR and consequently reduce the bit error rate (BER) of the FSO communication link. Furthermore, the proposed LSTM- and Transformer-based equalizers compensate for propagation-induced waveform distortion, detector bandwidth limitations, and residual pointing fluctuations after 100 m free-space transmission, thereby providing additional BER improvement.

3. Experimental Setup and Data Acquisition

Figure 2 shows the experimental FSO communication testbed used to evaluate the proposed system. As shown in Figure 2a, the transmitter (Tx) and receiver (Rx) are positioned on the same side of the experimental setup, while a high-reflectivity planar mirror is installed at the end of an indoor corridor. The optical beam emitted from the transmitter propagates over a distance of 50 m toward the mirror, is reflected back along the same optical path, and is finally collected by the receiver. Consequently, the total optical propagation distance is 100 m, corresponding to a double-pass FSO transmission link.
Figure 2. Experimental free-space optical communication testbed. (a) Photograph showing the indoor experimental setup. (b) Schematic of the optical transmission system, where the transmitted beam propagates 50 m toward a planar mirror and returns along the same optical path, resulting in a total optical propagation distance of 100 m. (c) Experimental measurement platform used for waveform generation, data acquisition, and signal processing.
The optical configuration of the proposed system is illustrated in Figure 2b. A laser source generates the optical carrier, which is first collimated using a collimation lens (Thorlabs, Newton, NJ, USA). The collimated beam then passes through a vortex plate, which converts the incident Gaussian beam into the desired orbital angular momentum (OAM) mode. The generated OAM beam is subsequently expanded and collimated using a Cassegrain telescope (Celestron, Torrance, CA, USA) to reduce beam divergence during free-space propagation. After propagating over 50 m, the beam is reflected by a planar mirror and returns along the same optical path. The reflected beam is collected by a Cassegrain telescope and directed to the receiver for optical detection and performance evaluation.
The optical parameters of both telescopes are summarized in Table 2. The clear aperture and secondary obstruction diameters were measured directly on the instruments, as the manufacturer specifications refer to the secondary mirror rather than to the full opaque assembly that blocks the beam.
Table 2. Optical parameters of the transmitting and receiving Cassegrain telescopes.
The experimental measurement platform is shown in Figure 2c. An arbitrary waveform generator (AWG70002A, Tektronix, Beaverton, OR, USA) generates the electrical modulation waveform for the optical transmitter, while a digital oscilloscope (DSO6054A, Santa Rosa, Keysight, CA, USA) acquires the received waveform for offline digital signal processing. A pulse arbitrary noise generator (81150A, Santa Rosa, Keysight, CA, USA) provides controlled electrical noise when channel impairment emulation is required. All instruments are synchronized and controlled by a laptop, which also performs waveform generation, data acquisition, and BER analysis.
The folded optical configuration enables long-distance FSO transmission measurements within a limited indoor environment while maintaining stable optical alignment. In addition, the double-pass propagation scheme provides an effective optical propagation distance of 100 m using a 50 m physical corridor, allowing the transmission performance of the proposed OAM-based FSO communication system to be evaluated under practical indoor conditions.
The optical coupling efficiency of the transmitted beam is evaluated by measuring the optical power at the output of the receiving Cassegrain telescope. A conventional Gaussian beam ( l = 0 ) and a vortex beam with a topological charge of l = 2 are measured under identical alignment and transmission conditions. The optical power is recorded using an optical power meter positioned immediately after the telescope eyepiece.
The FSO communication experiment is conducted over a 100 m horizontal free-space propagation path. The transmitted beam propagates through the optical channel and is collected by the receiving Cassegrain telescope. The received optical signal is focused onto an avalanche photodiode (APD), where the optical signal is converted into an electrical waveform for subsequent acquisition and analysis.
For communication performance evaluation, an on–off keying (OOK) modulation format is employed owing to its simple implementation and compatibility with intensity-modulation/direct-detection (IM/DD) systems. The detected electrical waveform is acquired using a high-speed digital oscilloscope (DSO6054A, Santa Rosa, Keysight, CA, USA) and transferred to a computer for offline digital signal processing. The acquired waveform is processed to obtain symbol-spaced sequences, which are subsequently used as input data for the Transformer and LSTM equalizers.
OOK was selected as a representative low-complexity IM/DD modulation format so that the effects of aperture matching and sequence equalization could be isolated without introducing an additional modulation-format variable. The data rate was selected near the transition region of the experimental system’s bandwidth. A received-power variation would change the SNR and absolute BER of all receivers but would not remove the temporal channel memory that distinguishes memoryless threshold detection from sequence equalization. The measurements were performed under the best achievable alignment to isolate the aperture-matching effect. A comprehensive evaluation of power, SNR, data rate, modulation format, and controlled misalignment is beyond the present scope and will be addressed in future work.
The acquired electrical waveform is converted into supervised learning samples using a causal sliding-window approach. Since the received waveform is sampled continuously by the oscilloscope, each input sequence consists of the current sample together with the preceding sixteen samples, resulting in a fixed sequence length of 17 samples. To preserve the temporal correlation of the communication channel and prevent information leakage, the received waveform is divided chronologically without random shuffling. The first 40% of the acquired data is used for training, the subsequent 10% is used for validation during hyperparameter optimization, and the remaining 50% is used for testing. The identical chronological partitioning strategy is applied to both the Transformer and LSTM equalizers to ensure a fair performance comparison.
The acquired sequence contains 4092 bits, corresponding to 1636 bits (40%) for training, 409 bits (10%) for validation, and the remaining 2047 (50%) bits for evaluation. The computational complexity is evaluated using the number of multiply–accumulate (MAC) operations, which provides a hardware-independent measure of neural network computation. The training complexity is expressed as the estimated number of GMACs required for one training epoch, whereas the inference complexity is expressed as the number of KMACs required to detect one transmitted bit. For the Transformer equalizer, the approximate inference complexity is expressed as
C T r , i n f = T F d + 4 T d 2 + 2 T 2 d + 2 T d d F F + 2 d ,
where T denotes the input sequence length, F denotes the number of input features, d = d m o d e l is the Transformer model dimension, and d F F is the feed-forward network dimension. The first term represents the input projection, the second term corresponds to the query, key, value, and output projections of the self-attention module, the third term is attention-score and weighted-value products, the fourth term accounts for the two feed-forward layers, and the last term represents the binary classification layer. For the LSTM equalizer, the inference complexity is expressed as
C L S T M , i n f = 4 T H F + H 2 + 2 H ,
where H denotes the number of hidden units. The factor of four accounts for the input, forget, cell-update, and output gates of the LSTM architecture. The computational cost required for network training is approximated as
C t r a i n ≈ 3 N t r C i n f ,
where N t r denotes the number of training windows. The factor of three approximates the combined computational cost of forward propagation, back-propagation, and parameter updating. It is noted that the reported computational complexity is intended only for architecture comparison and excludes hardware-dependent factors, such as memory access, activation-function evaluation, software framework overhead, and device-specific optimization. Therefore, the reported GMAC/epoch and KMAC/bit values represent hardware-independent computational estimates rather than actual execution time. Although these estimates omit hardware-dependent factors, the difference between the two architectures reported in Section 4.2 is large enough that the comparison is unlikely to be reversed by them. The difference is structural rather than incidental, since the attention operation scales as T 2 d with the sequence length T , whereas the recurrent update scales linearly in T ; memory traffic and framework overhead would have to differ by a comparable factor in the opposite direction to alter the ordering.

4. Results and Discussion

4.1. Experimental Verification of Aperture-Matched Coupling

To verify the proposed aperture-matching concept, the telescope-output beam profiles and received optical powers are experimentally compared for conventional Gaussian coupling and OAM-matched annular coupling. The beam patterns are recorded immediately after the receiving Cassegrain telescope under identical alignment conditions, and the corresponding optical powers are independently measured using an optical power meter (PM400, Thorlabs, Newton, NJ, USA).
Figure 3 compares the horizontal beam profiles extracted from the measured annular beam images together with the corresponding received optical powers. As shown in Figure 3a, the Gaussian beam exhibits a weaker annular intensity after propagation through the Cassegrain telescope because a significant fraction of the optical power is blocked by the secondary mirror. In contrast, the OAM-matched beam redistributes its optical energy toward an annular ring that more closely overlaps the unobstructed region of the telescope pupil.
Figure 3. Comparison of Gaussian and OAM-matched annular coupling at the output of the Cassegrain telescope. (a) Horizontal red-beam intensity profiles extracted across the central obscuration along the horizontal direction, with inset images showing the measured telescope-output beam patterns. The gray background and the central obscured region were assigned zero intensity, and both profiles were normalized to the peak intensity of the OAM-matched case. (b) Measured received optical power for Gaussian and OAM-matched coupling. The OAM-matched annular beam increased the mean received power from 0.603 mW to 0.810 mW, corresponding to a 34.3% improvement over Gaussian coupling. Error bars indicate ± 1 standard deviation over three independent measurement pairs.
For quantitative comparison, only the red annular beam region is regarded as the effective optical intensity during profile extraction, whereas the gray background and the central obscured region are assigned zero intensity. Consequently, the plotted profiles represent only the optical field contributing to effective coupling through the annular aperture.
Although the beam profiles are normalized to the peak intensity of the OAM-matched beam to emphasize the spatial intensity distribution, the independently measured optical-power results shown in Figure 3b confirm the corresponding coupling improvement. The average received optical power increases from 0.603 mW for Gaussian coupling to 0.810 mW for the OAM-matched beam, corresponding to a 34.3% enhancement. The values are the means of three repetitions in which the spiral phase plate was alternately inserted and removed under a fixed telescope alignment, with standard deviations of 0.095 mW and 0.085 mW, respectively. Since the two conditions form a pair at each repetition, the variation introduced by exchanging the plate cancels in the comparison. The improvement is consistent across all three pairs, with power differences of 0.22, 0.20 and 0.20 mW, and a paired t-test gives t = 31.0 with p = 0.001, whereas a Welch test ignoring the pairing gives p = 0.049. The mean difference of 0.207 mW has a 95% confidence interval of [0.178, 0.235] mW, so even the lower bound corresponds to a 29.5% improvement. Both conditions were measured with the same power meter at the same wavelength, so the instrument calibration uncertainty cancels in the ratio.
The measured improvement can be related to the overlap model of Section 2. From the profiles of Figure 3a, recorded at the entrance aperture of the receiving telescope, the annular intensity maximum of the OAM-matched beam lies at 38.4 mm from the axis, within the unobstructed annulus extending from 17.5 mm to 63.5 mm, so the matching criterion of Equation (8) is satisfied and the measured ring radius agrees with Equation (6) at l = 2 . Equation (9) evaluated for an ideal Laguerre–Gaussian field of this ring radius gives a coupling efficiency exceeding the measured improvement, as expected, since the transmitted field is a Gaussian beam modified by a spiral phase plate rather than a pure Laguerre–Gaussian mode and the measured ring is not azimuthally uniform, with peak radii of 42.0 mm and 34.8 mm on opposite sides. The model therefore serves as a design criterion for placing the ring within the annulus rather than as a predictor of the absolute coupling gain.
These results experimentally validate the aperture-matching principle introduced in Section 2. By redistributing the optical energy away from the optical axis, the vortex beam reduces obstruction-induced loss and increases the overlap between the transmitted optical field and the unobstructed annular pupil of the Cassegrain telescope. The overlap model of Section 2 assumes an ideal Laguerre–Gaussian field and perfect centering and therefore represents an upper bound; the measured improvement is smaller because the spiral phase plate does not produce a pure Laguerre–Gaussian mode and residual on-axis intensity and alignment error remain.
Although the proposed aperture-matched vortex beam significantly improves the received optical power, waveform distortion caused by the limited modulation bandwidth and residual inter-symbol interference (ISI) still remains after optical transmission. Therefore, deep learning-based sequence equalizers are introduced in the following subsection to compensate for these impairments and further improve the communication performance.

4.2. Deep Learning-Based Signal Recovery

To determine suitable neural network architectures for waveform recovery, both the Transformer and LSTM equalizers are optimized by jointly considering BER performance and computational complexity. The computational cost is estimated using the MAC-based complexity models presented in Section 3.
Figure 4a presents the optimization results of the Transformer equalizer. Increasing the model dimension generally reduces the BER because the self-attention mechanism captures longer temporal dependencies introduced by channel memory. However, the corresponding computational complexity also increases rapidly, indicating a clear trade-off between recovery performance and hardware cost. Although larger Transformer models achieve lower BER values, the performance improvement gradually saturates while the required GMAC per training epoch continues to increase.
Figure 4. Machine learning-based equalizer architecture optimization and recovery performance for the 100-kbit/s OAM-coupled free-space optical signal. (a) Test BER and estimated training complexity for a proportionally scaled Transformer family, where the head dimension and the feed-forward dimension are set optimally. (b) Test BER and estimated training complexity as functions of the LSTM hidden-unit count after fixing the dropout probability and learning rate at their coarse-search optima. (c) BER comparison among the optimized decision-threshold detector, Transformer, and LSTM receivers. The LSTM produced no observed errors over the evaluated test sequence and is therefore plotted at the display floor of 10 − 4 . The annotated values represent architecture-based training and inference operation counts. (d) Symbol-spaced eye diagrams for the threshold, Transformer, and LSTM outputs. The bandwidth-limited threshold signal exhibits a nearly closed eye due to severe intersymbol interference, whereas both sequence-learning receivers substantially reopen the decision eye.
Figure 4b shows the optimization results of the LSTM equalizer. Increasing the number of hidden units initially improves the BER because the recurrent hidden state effectively preserves temporal information across consecutive received symbols. The minimum BER is achieved with approximately 15 hidden units, whereas additional hidden units provide only marginal performance improvement while increasing computational complexity. Compared with the Transformer, the LSTM achieves comparable or superior recovery performance with significantly lower computational cost.
The final communication performance is summarized in Figure 4c. The conventional threshold detector produces a BER of 0.179, demonstrating that symbol-by-symbol detection cannot adequately compensate for the severe waveform distortion produced by the bandwidth-limited transmitter. The optimized Transformer reduces the BER to 0.0133, corresponding to an improvement of approximately ten-fold compared with threshold detection. The optimized LSTM produces no bit errors over the 2047-bit evaluation sequence. With zero observed errors, the bit error rate is bounded above by 1.5 × 10 − 3 at 95% confidence, and this bound, rather than an exact zero, is the quantity the measurement establishes.
For reference, conventional equalizers were applied to the same data under the identical partition and window length. A minimum mean square error linear equalizer reduces the bit error rate to 0.0298 and a decision-feedback equalizer to 0.0195, so that a large part of the loss incurred by threshold detection is recoverable at negligible computational cost; nonetheless, the machine learning-based equalizers outperform these by recovering residual distortion not captured by these linear or piecewise-linear models.
The corresponding eye diagrams are presented in Figure 4d. The threshold detector exhibits an almost completely closed eye because of severe inter-symbol interference, whereas both neural equalizers substantially reopen the eye. Among the evaluated methods, the LSTM provides the widest eye opening and the clearest symbol separation, demonstrating its superior capability to compensate for channel memory and residual ISI.
Overall, the proposed communication system benefits from two complementary mechanisms. The aperture-matched vortex beam first improves the optical coupling efficiency by reducing the loss caused by the central obscuration of the Cassegrain telescope, thereby increasing the received optical power. The neural equalizer subsequently compensates for the remaining temporal distortion introduced by the bandwidth-limited transmitter and receiver. While both deep learning models significantly improve the communication performance, the LSTM provides the most favorable balance between BER reduction and computational complexity, making it the preferred equalizer for the proposed FSO communication system.

4.3. Scalability to Longer Link Distances

The present experiment uses a folded 100 m path, for which L ≪ z R and the annular geometry at the receiver is essentially identical to that at the transmitter. The expansion remains moderate over several kilometers and can be compensated by co-designing w 0 and l with the intended link length or by increasing the transmitter aperture.
Turbulence is the more restrictive limitation. Since the Rytov variance scales as L 11 / 6 , turbulence that is negligible over the present indoor path becomes significant at the kilometer scale. Turbulence-induced degradation of OAM mode purity does not directly affect the present scheme, because the vortex beam is used as aperture-matched illumination rather than as a preserved eigenmode, and the receiver performs direct detection of the collected power. Beam wander, in contrast, displaces the ring relative to the central obscuration and acts directly on the coupling mechanism and must be absorbed by the margin ∆ r or mitigated by tracking.
The distortion compensated by the equalizers in Section 4.2 originates from the transmitter modulation bandwidth and is independent of link length, so the equalization gain is expected to transfer, although kilometer-scale operation would require online adaptation to track turbulence-induced fading. The folded 100 m configuration was chosen deliberately for this reason: it places the link in a regime where both diffractive spreading and turbulence are negligible, so that the coupling improvement can be attributed to the annular matching itself rather than to propagation conditions that vary between measurements.

5. Conclusions

This paper proposed a vortex beam-assisted FSO communication system that combines aperture-matched optical coupling with deep learning-based signal recovery for centrally obscured Cassegrain telescopes. An annular aperture coupling model was presented to analyze the influence of the secondary-mirror obstruction on optical transmission, and the corresponding optical coupling characteristics were experimentally verified using a 100 m FSO link consisting of Cassegrain transmitter and receiver front-ends.
The experimental results demonstrated that the annular intensity distribution of the vortex beam more effectively matches the unobstructed aperture of the telescope than a conventional Gaussian beam, thereby increasing the received optical power from 0.603 mW to 0.810 mW, corresponding to a 34.3% improvement in optical coupling efficiency. To compensate for waveform distortion caused by the limited modulation bandwidth and residual inter-symbol interference, LSTM- and Transformer-based sequence equalizers were investigated. Both neural network equalizers significantly improved the communication performance compared with conventional threshold detection, while the optimized LSTM equalizer achieved the lowest BER with lower computational complexity than the Transformer.
Overall, the proposed system demonstrates that combining aperture-matched vortex beam transmission with deep learning-based equalization provides an effective solution for improving both optical coupling efficiency and communication reliability in compact telescope-assisted FSO systems. The aperture-matching mechanism demonstrated here applies wherever a centrally obscured telescope is used as an optical front end, and the equalization gain originates from the transmitter modulation bandwidth and is therefore independent of link length. Extending the present results to ground-to-UAV or satellite links would additionally require the effects of atmospheric turbulence, pointing and tracking error, and the associated propagation loss to be established, none of which are evaluated in this work. Future work will investigate longer transmission distances under atmospheric turbulence and real-time implementation of neural network equalizers for practical FSO communication systems. A further direction of interest is the extension of the aperture-overlap analysis of Section 2 to structured fields carrying multiple phase singularities, which offer additional control over the transverse intensity distribution but are not circularly symmetric about the beam axis.

Author Contributions

Conceptualization, J.B. and H.C.; methodology, Y.L.; software, J.B.; validation, Y.L., J.B. and H.C.; formal analysis, J.B.; investigation, Y.L.; data curation, Y.L.; writing—original draft preparation, J.B.; writing—review and editing, Y.L.; visualization, J.B.; supervision, H.C.; project administration, H.C.; funding acquisition, H.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the MSIT, Republic of Korea, under the ITRC support program (IITP-2026-RS-2023-00259061) supervised by the IITP. Also, this work was supported by the National Research Foundation of Korea (NRF) grant funded by MSIT (No. RS-2026-25486449).

Data Availability Statement

Dataset available on request from the authors.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Al-Gailani, S.A.; Mohd Salleh, M.F.; Salem, A.A.; Shaddad, R.Q.; Sheikh, U.U.; Algeelani, N.A.; Almohamad, T.A. A Survey of Free Space Optics (FSO) Communication Systems, Links, and Networks. IEEE Access 2021, 9, 7353–7373. [Google Scholar] [CrossRef] [Scilit]
  2. Khalighi, M.A.; Uysal, M. Survey on Free Space Optical Communication: A Communication Theory Perspective. IEEE Commun. Surv. Tutor. 2014, 16, 2231–2258. [Google Scholar] [CrossRef] [Scilit]
  3. Chaudhry, A.U.; Yanikomeroglu, H. Free Space Optics for Next-Generation Satellite Networks. IEEE Consum. Electron. Mag. 2021, 10, 21–31. [Google Scholar] [CrossRef] [Scilit]
  4. Lian, Y.; Qi, X.; Wang, Y.; Bai, Z.; Wang, Y.; Lu, Z. OAM beam generation in space and its applications: A review. Opt. Lasers Eng. 2022, 151, 106923. [Google Scholar] [CrossRef] [Scilit]
  5. Allen, L.; Beijersbergen, M.W.; Spreeuw, R.J.C.; Woerdman, J.P. Orbital angular momentum of light and the transformation of Laguerre-Gaussian laser modes. Phys. Rev. A 1992, 45, 8185–8189. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  6. Ji, W.; Lee, C.-H.; Chen, P.; Hu, W.; Ming, Y.; Zhang, L.; Lin, T.-H.; Chigrinov, V.; Lu, Y.-Q. Meta-q-plate for complex beam shaping. Sci. Rep. 2016, 6, 25528. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  7. Guo, M.; Le, W.; Wang, C.; Rui, G.; Zhu, Z.; He, J.; Gu, B. Generation, Topological Charge, and Orbital Angular Momentum of Off-Axis Double Vortex Beams. Photonics 2023, 10, 368. [Google Scholar] [CrossRef] [Scilit]
  8. Chen, L.; Wang, G.; Yin, Y.; Zhong, H.; Liu, D.; Wang, Y. Partially Coherent Off-Axis Double Vortex Beam and Its Properties in Oceanic Turbulence. Photonics 2024, 11, 20. [Google Scholar] [CrossRef] [Scilit]
  9. Chen, L.; Liu, D.; Gao, H.; Dong, A.; Wang, Y. Research on characteristics of partially coherent radially polarized off-axis double vortex beam in oceanic turbulence. Opt. Laser Technol. 2025, 190, 113264. [Google Scholar] [CrossRef] [Scilit]
  10. Manavalan, G.; Balasubramaniam, G.M.; Arnon, S. Improving OAM-Based Optical Wireless Communication in Turbulence Using Conjugate Light Field and Hybrid Neural Networks. J. Light. Technol. 2025, 43, 3211–3221. [Google Scholar] [CrossRef] [Scilit]
  11. Jiang, Z.; Su, X.; Zhou, H.; Zuo, Y.; Duan, Y.; Ramakrishnan, M.; Alhaddad, A.; Lian, H.; Wang, Y.; Zeng, R.; et al. Experimental Demonstration of Using a Low-Divergence Pin-Like Beam in a 2-Gbit/s FSO Link With Limited Receiver Aperture Sizes and Variable Link Lengths Under Turbulence Effects. J. Light. Technol. 2026, 44, 1330–1336. [Google Scholar] [CrossRef] [Scilit]
  12. Hu, N.; Zhou, H.; Zhang, R.; Song, H.; Pang, K.; Zou, K.; Song, H.; Su, X.; Liu, C.; Lynn, B.; et al. Experimental demonstration of a “pin-like” low-divergence beam in a 1-Gbit/s OOK FSO link using a limited-size receiver aperture at various propagation distances. Opt. Lett. 2022, 47, 4215–4218. [Google Scholar] [CrossRef] [Scilit]
  13. Xiao, L.; Yang, H.; Liu, X.; Xu, Q.; Xiong, X. New type optical Cassegrain antenna with lenses telescope system. Optik 2010, 121, 521–525. [Google Scholar] [CrossRef] [Scilit]
  14. Wang, J.; Li, M.; Han, X.; Tan, Y.; Li, H.; Lin, X. Hollow Gaussian beam-based experimental investigation on echo measurements under atmospheric turbulence and central obstruction. Opt. Express 2024, 32, 30702–30712. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  15. Ke, X.; Lei, S. Spatial light coupled into a single-mode fiber by a Maksutov-Cassegrain antenna through atmospheric turbulence. Appl. Opt. 2016, 55, 3897–3902. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  16. Kang, H.J.; Yang, J.; Chun, B.J.; Jang, H.; Kim, B.S.; Kim, Y.-J.; Kim, S.-W. Free-space transfer of comb-rooted optical frequencies over an 18 km open-air link. Nat. Commun. 2019, 10, 4438. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  17. Yu, M.; Jiang, P.; Yang, H.; Huang, Y.; Ma, X.; Wang, B. Improving transmission efficiency of Cassegrain antenna. Optik 2015, 126, 795–798. [Google Scholar] [CrossRef] [Scilit]
  18. Wu, H.; Wu, W.; Xu, X.; Chen, J.; Zhao, Y. A new method to improve power efficiencies of optical systems with Cassegrain-telescope receivers. Opt. Commun. 2011, 284, 3361–3364. [Google Scholar] [CrossRef] [Scilit]
  19. Wu, H.; Sheng, S.; Huang, Z.; Zhao, S.; Wang, H.; Sun, Z.; Xu, X. Study on power coupling of annular vortex beam propagating through a two-Cassegrain-telescope optical system in turbulent atmosphere. Opt. Express 2013, 21, 4005–4016. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  20. Lee, J.-Y.; Baek, J.; Kim, J.; Rajbhandari, S.; Park, S.R.; Chun, H. Concentric Intensity-Based Adjacent OAM Mode Separation for High-Efficiency Free-Space Optical Spatial Multiplexing. Appl. Sci. 2025, 15, 8949. [Google Scholar] [CrossRef] [Scilit]
  21. Kupferman, J.; Arnon, S. Direct detection receiver for vortex beam. J. Opt. Soc. Am. A 2018, 35, 1543–1548. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  22. Aksenov, V.P.; Izmailov, I.V.; Kanev, F.Y.; Poizner, B.N. Optical vortex detector as a basis for a data transfer system: Operational principle, model, and simulation of the influence of turbulence and noise. Opt. Commun. 2012, 285, 905–928. [Google Scholar] [CrossRef] [Scilit]
  23. Cao, M.; Yao, R.; Xia, J.; Jia, K.; Wang, H. LSTM Attention Neural-Network-Based Signal Detection for Hybrid Modulated Faster-Than-Nyquist Optical Wireless Communications. Sensors 2022, 22, 8992. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  24. Palitharathna, K.W.S.; Wickramasinghe, N.D.; Vegni, A.M.; Suraweera, H.A. Neural Network-Based Optimization for SLIPT-Enabled Indoor VLC Systems With Energy Constraints. IEEE Trans. Green Commun. Netw. 2024, 8, 839–851. [Google Scholar] [CrossRef] [Scilit]
  25. Hemmati, H.; Page, N. Efficient Coupling of Lasers to Telescopes with Obscuration; Report No. NPO-30574; NASA Tech Briefs: New York, NY, USA, 2003. Available online: https://ntrs.nasa.gov/citations/20110024095 (accessed on 1 July 2026).
  26. Kogelnik, H.; Li, T. Laser beams and resonators. Appl. Opt. 1966, 5, 1550–1567. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  27. Vallone, G. Role of beam waist in Laguerre-Gauss expansion of vortex beams. Opt. Lett. 2017, 42, 1097–1100. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  28. Liu, T.; Tian, M.; Sheng, Q.; Fu, S.; Shi, W.; Yao, J. Research on the intensity profiles of high-order Laguerre-Gaussian mode laser beams. Infrared Phys. Technol. 2024, 141, 105459. [Google Scholar] [CrossRef] [Scilit]
  29. Tschunko, H.F.A. Imaging Performance of Annular Apertures. Appl. Opt. 1974, 13, 1820–1823. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  30. Stamnes, J.J.; Heier, H.; Ljunggren, S. Encircled energy for systems with centrally obscured circular pupils. Appl. Opt. 1982, 21, 1628–1633. [Google Scholar] [CrossRef] [Scilit] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.