A Deep Reinforcement Learning Algorithm for Smart Control of Hysteresis Phenomena in a Mode-Locked Fiber Laser

Abstract

We experimentally demonstrate the application of a double deep Q-learning network algorithm (DDQN) for design of a self-starting fiber mode-locked laser. In contrast to the static optimization of a system design, the DDQN reinforcement algorithm is capable of learning the strategy of dynamic adjustment of the cavity parameters. Here, we apply the DDQN algorithm for stable soliton generation in a fiber laser cavity exploiting a nonlinear polarization evolution mechanism. The algorithm learns the hysteresis phenomena that manifest themselves as different pumping-power thresholds for mode-locked regimes for diverse trajectories of adjusting optical pumping.

1. Introduction

Machine learning (ML) algorithms have already shown their potential for improving performance of ultrafast photonic systems [1]. In particular, applications of ML in mode-locked fiber lasers target three main areas: self-starting [2], system optimization [3] and characterization [4]. Most of the conventional ML methods optimize static topology of the fitness function and do not take into account the trajectory of adjusting controlling parameters [5,6,7]. However, fiber mode-locked lasers are complex nonlinear systems that are sensitive to initial conditions and often possess hysteresis phenomena [8,9] that require special approaches to smart control of such devices. Hysteresis and bistability effects may lead to incorrect operation of the optimization algorithms—e.g., genetic or particle swarm optimization algorithms that do not consider dynamic changes in laser cavity parameters. For given control parameters and different tuning strategies, the algorithm will meet different output regimes with corresponding different values of the optimized fitness function. Reinforcement learning (RL) algorithms are attractive candidates for dynamic optimization [10]. In the context of mode-locked lasers, there are already promising applications of deep RL to control the output radiation. Kutz et al. [11] demonstrated using the numerical model of fiber cavity that a deep Q-learning algorithm is capable of learning how to operate with bi-stability to achieve stable mode-locking regimes. In [12], the possibility of stabilizing the mode-locking operation of an NPE laser under temperature and vibratory disturbances by an actor–critic RL algorithm was demonstrated. In our previous work, we have showed that the DDQN algorithm may be efficiently applied to a changeable environment, such as an 8-figure mode-locked laser with a tunable spectral filter [13].

In this Letter we experimentally validate RL’s ability to find nontrivial optimization trajectories when adjusting pumping power of the laser cavity for soliton generation with targeted properties.

2. Experimental Setup

We consider here a fiber mode-locked laser based on the nonlinear polarization effect as an experimental platform; see Figure 1. The laser comprises an active Erbium fiber, an electronically driven polarization controller (EPC), an output coupler, a piece of passive SMF-28 fiber and an optical isolator with blocked fast axis. Active fiber was pumped through the WDM coupler by a single-mode laser diode operating at 978 nm. The fiber laser cavity was placed on a thermal plate inside a thermostat box to eliminate the influence of the ambient temperature fluctuations. The temperature of the plate was controlled by a PID regulator and was set to 32 °C. The goal for the RL algorithm was to learn the behavior of the laser system and to obtain a stable mode-locked regime at a low level of pumping power. We deliberately chose such a relatively simple setup to be able to comprehensively verify the correctness of the algorithm and to demonstrate the feasibility of the method.

Anomalous dispersion of the laser cavity provides the conditions for generation of classical optical solitons. A stable mode-locked regime was achieved by proper adjustment of the EPC controlling voltage and reaching the threshold value of the pumping power. The repetition rate of a pulse train was 14.51 MHz with an energy of 1.2 pJ for the pulses. Figure 2a depicts an optical spectrum of the generated solitons with the characteristic Kelly side-bands. A further increase in the pumping power led to multi-soliton generation. For instance, single-soliton generation was obtained at 290 mA (Figure 2b); then, after reaching 310 mA, the output switched to double-soliton regime.

Multi-soliton generation (also known as soliton quantization) features a hysteresis effect: the number of solitons is different for different directions of adjusting the pumping power [14,15,16]. Starting from the high level of pumping power, stable soliton generation remained until 280 mW, but it did not exist in case of starting from the low level of the pumping power (Figure 2b). Initiation of a mode-locked lasing regime requires more pumping power than supporting stable soliton generation when it already exists. The goal for the reinforcement learning algorithm was to learn this hysteresis phenomenon without explicit programming and obtain a stable soliton regime at the lowest level of pumping power.

3. Reinforcement Learning

The RL algorithm operates with the mode-locked laser as a black-box. It does not have information about underlying physics and learns the behavior of the system through the actions and their consequences. In terms of the RL-algorithm, the considered problem is formulated as follows: the laser is an environment where an agent, a neural network, is acting by changing the parameters of the cavity. Fixed parameters of the laser cavity determine the characteristics of the output optical radiation, which are denoted as a state of the environment. The goal for the agent is to maximize the reward gained for the appropriate actions. Reward r is a function that is designed to have maximum at desired mode-locked regime. In our case, we have used the following function:

$$r=\frac{1}{max\left(ACF\right)}$$

(1)

where $max\left(ACF\right)$ is the amplitude of the maximum central peak at the autocorrelation function of the oscilloscope trace. This function provides the highest reward for mode-locked regimes at the lowest pumping power levels. We also have found that the autocorrelation function of an oscilloscope trace provides a convenient way to filter unstable, q-switched and continuous wave regimes (Figure 3). At the first stage, the autocorrelation function of the oscilloscope trace is calculated. Then, the algorithm detects the number of the peaks applying the peak detection algorithm available through Python Scipy package. If the number of the peaks is less than 2, the reward is zero; otherwise, the reward is calculated with Equation (1).

Reinforcement learning algorithms usually require a large amount of training time. In case of an experimental mode-locked laser, there is no straightforward possibility to parallel the training procedure. We have used the double deep Q-learning network (DDQN) as a solution with a balanced trade-off between the stability and training time. For instance, the less complicated deep Q-learning algorithm suffers from an overestimation effect [17], and the actor–critic algorithm requires more training time and is less data efficient.

The scheme of the applied DDQN algorithm is depicted at Figure 4a. Firstly, the feed-forward neural network $Q_{net}$ is initialized with random weights of neurons. $Q_{net}$ predicts the action that has to be done based on the current state. The state in our case is a combination of laser diode current and reward that was measured at this current. The input layer has two neurons; the first corresponds to the the current of the laser diode, and the second—to the estimated reward. The output layer has two neurons that correspond to the actions: increasing or decreasing the current of the pumping laser diode by 1 mA. $Q_{net}$ also contains three hidden fully connected layers with 32 neurons each with a ReLu activation function. Next, $Q_{tnet}$ is initialized by copying $Q_{net}$. Due to the low dimensions of the input and output layers, we did not use large neural networks. The final architectures of neural networks were designed by optimization of their performances to the toy numerical model of the hysteresis effect.

To prevent oscillation and divergence of the algorithm, we implemented the concept of experience replay buffer [18]. It breaks data correlation and makes it possible to use old experience for off-policy algorithms [19]. An experience replay buffer is a table containing the tuple of initial state, action, final state and reward that is obtained by transferring from initial to final state after the action. After $Q_{net}$ and $Q_{tnet}$ initialization, an experience replay buffer with a fixed size of 1000 was filled by records of acting based on $Q_{net}$ predictions. Finally, we introduced the penalty procedure to avoid the situation when the agent receives high cumulative reward by oscillating around some intermediate point at the hysteresis curve. After each action, 10% of the maximum possible reward is deducted from the total reward. The training procedure includes the stage during which the agent makes 100 actions from initial point, trying to maximize the cumulative reward. Every action, the experience buffer was updated by substituting the old records by a new ones. The experience buffer was used as a training dataset for $Q_{net}$. The weights of $Q_{net}$ were updated every 3 actions, and after 50 actions, the weights of $Q_{tnet}$ were copied from $Q_{net}$. The new weights were calculated by backpropagation with an adam optimizer [20] at a learning rate of ${10}^{-4}$. The optimizer minimizes the following loss function:

$$L={\left(r+\gamma \underset{a^{\prime }}{max}{Q}_{tnet}\left(s^{\prime },a^{\prime }\right)-Q_{net}\left(s,a\right)\right)}^2$$

(2)

where $\gamma$—discount factor equal to 0.9; s, $s^{\prime }$ —current and next states of the environment; a—action. This loss function allows one to find the optimal Q-function which maximizes the cumulative reward during the self-adjusting procedure [21]. The Q-function is defined as the expected discounted reward for an agent making action a from a state s and then following policy $\pi$. Mathematically, it can be described in the following way:

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[R_t\right|s_t=s,a_t=a]$$

(3)

where $R_t$ is the expected reward that we will receive at the end of the episode. Using two neural networks allows one to solve overestimation bias for predicting the value of the Q-function. To determine the value of the Q-function, the minimum of the predictions of both networks is used. In order to run parallel learning of two networks asynchronously, two identical networks $Q_{net}$ and $Q_{tnet}$ were used. $Q_{net}$ was trained by a back propagation method, and $Q_{tnet}$ copied the weights of $Q_{net}$ every few steps.

To make the agent act different during the learning process, thereby exploring the environment, an $ϵ$-greedy strategy was used to select an action. During the episode, the agent choose an action predicted by $Q_{net}$ network with probability $(1-ϵ)$ or acted randomly with probability $ϵ$, where $ϵ$ starts from value 0.5. Each epoch consisted of 10 episodes $ϵ$ multiplied by 0.99. The iterations of the algorithm ran while the probability to make random actions was higher than ${10}^{-4}\%$. Figure 4b depicts the learning curves of the DDQN algorithms. After each epoch, we tested the trained agent on a laser system and measured the maximum reward it obtained after 100 actions.

There are three key stages of the agent’s learning process. First, the agent makes almost random steps and does not explore the mode-locked regime. At this first stage the agent gets almost zero or negative reward. At second stage, the agent finds the mode-locked regime at a high level of pumping power, but then, the actions are close to random. At the last stage, the agent finds the way to decrease the pumping power to increase the reward. One may mention the corresponding three levels with around 0, 2.5 and 8.5 values for the reward. The abrupt drops in the reward at the last stage of the learning curve are caused by excessive step-down of the agent, which leads to breaking of a mode-locked regime. Since the experimental laser exhibits power fluctuations, the threshold value of the pumping power slightly changes. One experimental sampling point consisting of one step of pumping diode current and subsequent measurement of a oscilloscope trace took 0.48 s. Therefore, overall training time during 800 epochs was approximately 110 h.

Figure 4c depicts the strategy of the trained agent. The agent starts tuning a current of the laser diode from the initial point at 280 mA. Blue points correspond to increasing the current level by 1 mA, and red points correspond to decreasing the current level by 1 mA. At the first stage, the agent increases the current step-by-step until 294 mA—when the mode-locked regime emerges. Then, the agent starts to decrease the current while maintaining the mode-locked regime. The crucial point for successive adjustment is to stop decreasing the current on time. Figure 4c demonstrates that the agent has learned the threshold level of the current below which the mode-locked regime breaks up. On reaching the vicinity of the reward maximum, the agent consequently increases and decreases the current maintaining stable mode-locked regime at the lowest current.

4. Conclusions

We have demonstrated the feasibility of using the DDQN algorithm for the self-adjusting operation of a mode-locked fiber laser. The attractive feature of the algorithm is its ability to find a dynamic strategy to tune the pumping power of the laser diode in order to achieve a stable mode-locked regime. We would like to note that this is an initial demonstration and we expect to improve overall performance. For instance, at the moment, despite the relatively simple task, a large amount of training data was required to teach the agent. In the case of a high-dimensional task, the number of required training samples would increase dramatically. Therefore, we believe that further development of RL algorithms applied to experimental laser systems should be combined with the appropriate numerical models capable of generating synthetic data close to the experimental ones. This would enhance the approach through distributed learning [22] approaches and parallelization of the learning process. Additionally, the transfer learning [23,24,25] concept may be applied to speed up the learning process; for instance, the already trained network may be applied to other laser system with hysteresis phenomena. We anticipate that our work will stimulate further study of the dynamic design and optimization of complex laser systems, when it is necessary to determine not optimal parameters of the cavity, but also trajectories (in the space of parameters) of reaching them. We believe that results contribute to development of “smart” fiber lasers that can self-adjust their operation to the changing environment.