# Multi-Objective Evolutionary Architecture Search for Parameterized Quantum Circuits

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Preliminaries and Related Work

#### 2.1. Quantum Computation Preliminaries

#### 2.2. Parameterized Quantum Circuits

#### 2.3. Quantum Architecture Search

## 3. Method

#### 3.1. Genetic Algorithm for Quantum Architecture Search

- 1.
- Perform a non-dominated sorting in the population of quantum architectures and classify them according to an ascending level of non-domination based on objectives.
- 2.
- Use crowding distance, which is related to the density of solutions with similar objective metrics, to perform Crowding-sort that makes the population less dense.
- 3.
- Generate offspring using crowded tournament selection, then apply genetic operators such as mutation and crossover.

#### 3.2. Encoding Scheme and Search Space

- ${\mathbf{x}}_{1}$: Variational PQC—A circuit with single-qubit rotations ${R}_{x}$, ${R}_{y}$, ${R}_{z}$ (Equation (1)) performed on each qubit, with the rotation angles as trainable parameters. For generality, we consider qubit rotation in 3-dimensional space (i.e., applying ${R}_{x}$, ${R}_{y}$, ${R}_{z}$) for all the qubits. This operator is used to change the qubit states based on the trainable parameters.
- ${\mathbf{x}}_{2}$: Entanglement—A circuit that performs circular entanglement to all the qubits by applying one or multiple controlled-Z gates (Equation (2)). In this work, we only consider circular entanglement, which has been widely used in prior studies.
- ${\mathbf{x}}_{3}$: Data-encoding PQC—A circuit with single-qubit rotations ${R}_{x}\left(\theta \right)$ (Equation (1)) performed on each qubit, with the rotation angle $\theta $ being the input data d scaled by trainable parameters $\mathsf{\lambda}$,$$\theta =\sigma (\mathsf{\lambda}d)$$While prior work Ding and Spector [15] uses linear activation, we propose to use non-linear activations such as Tanh to allow effective stacks of consecutive ${\mathbf{x}}_{3}$ circuits, since linear activation leads to multiple consecutive rotations being equivalent to a single rotation (Equation (11)).
- ${\mathbf{x}}_{0}$: Measurement—A Variational PQC (${\mathbf{x}}_{1}$) with trainable parameters followed by measurement to obtain the qubit observables. The outcome is a binary value for each qubit with different probabilities. The outputs are computed by a linear weighting of the observables by another set of trainable parameters for each output, with optional activation functions, e.g., Softmax for action probabilities. The architecture encoding/decoding is terminated when approaching ${\mathbf{x}}_{0}$.

#### 3.3. Evolutionary Quantum Architecture Search

#### 3.4. Multi-objective Optimization for Quantum-Classical Systems

## 4. Experiments

#### 4.1. RL Environments

#### 4.2. Implementation Details

#### 4.3. Single-Objective Results

#### 4.4. Multi-Objective Results

#### 4.5. Pattern Analysis in Quantum Architectures

## 5. Discussion of Limitations and Broader Impacts

## 6. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Preskill, J. Quantum computing in the NISQ era and beyond. Quantum
**2018**, 2, 79. [Google Scholar] [CrossRef] - Benedetti, M.; Lloyd, E.; Sack, S.; Fiorentini, M. Parameterized quantum circuits as machine learning models. Quantum Sci. Technol.
**2019**, 4, 043001. [Google Scholar] [CrossRef] [Green Version] - Schuld, M.; Bocharov, A.; Svore, K.M.; Wiebe, N. Circuit-centric quantum classifiers. Phys. Rev. A
**2020**, 101, 032308. [Google Scholar] [CrossRef] [Green Version] - Havlíček, V.; Córcoles, A.D.; Temme, K.; Harrow, A.W.; Kandala, A.; Chow, J.M.; Gambetta, J.M. Supervised learning with quantum-enhanced feature spaces. Nature
**2019**, 567, 209–212. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Schuld, M.; Killoran, N. Quantum machine learning in feature hilbert spaces. Phys. Rev. Lett.
**2019**, 122, 040504. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Li, G.; Song, Z.; Wang, X. VSQL: Variational shadow quantum learning for classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual, 2–9 February 2021; Volume 35, pp. 8357–8365. [Google Scholar]
- Liu, J.G.; Wang, L. Differentiable learning of quantum circuit born machines. Phys. Rev. A
**2018**, 98, 062324. [Google Scholar] [CrossRef] [Green Version] - Zhu, D.; Linke, N.M.; Benedetti, M.; Landsman, K.A.; Nguyen, N.H.; Alderete, C.H.; Perdomo-Ortiz, A.; Korda, N.; Garfoot, A.; Brecque, C.; et al. Training of quantum circuits on a hybrid quantum computer. Sci. Adv.
**2019**, 5, eaaw9918. [Google Scholar] [CrossRef] [Green Version] - LaRose, R.; Coyle, B. Robust data encodings for quantum classifiers. Phys. Rev. A
**2020**, 102, 032420. [Google Scholar] [CrossRef] - Huang, H.L.; Du, Y.; Gong, M.; Zhao, Y.; Wu, Y.; Wang, C.; Li, S.; Liang, F.; Lin, J.; Xu, Y.; et al. Experimental quantum generative adversarial networks for image generation. Phys. Rev. Appl.
**2021**, 16, 024051. [Google Scholar] [CrossRef] - Jerbi, S.; Gyurik, C.; Marshall, S.; Briegel, H.; Dunjko, V. Parametrized Quantum Policies for Reinforcement Learning. Adv. Neural Inf. Process. Syst.
**2021**, 34, 28362–28375. [Google Scholar] - Skolik, A.; Jerbi, S.; Dunjko, V. Quantum agents in the gym: A variational quantum algorithm for deep q-learning. arXiv
**2021**, arXiv:2103.15084. [Google Scholar] [CrossRef] - Pérez-Salinas, A.; Cervera-Lierta, A.; Gil-Fuster, E.; Latorre, J.I. Data re-uploading for a universal quantum classifier. Quantum
**2020**, 4, 226. [Google Scholar] [CrossRef] [Green Version] - Elsken, T.; Metzen, J.H.; Hutter, F. Neural architecture search: A survey. J. Mach. Learn. Res.
**2019**, 20, 1997–2017. [Google Scholar] - Ding, L.; Spector, L. Evolutionary quantum architecture search for parametrized quantum circuits. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, Boston, MA, USA, 9–13 July 2022; pp. 2190–2195. [Google Scholar]
- Lu, Z.; Whalen, I.; Boddeti, V.; Dhebar, Y.; Deb, K.; Goodman, E.; Banzhaf, W. Nsga-net: Neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference, Prague, Czech Republic, 13–17 July 2019; pp. 419–427. [Google Scholar]
- Deb, K. Multi-objective optimisation using evolutionary algorithms: An introduction. In Multi-Objective Evolutionary Optimisation for Product Design and Manufacturing; Springer: London, UK, 2011; pp. 3–34. [Google Scholar]
- Helmuth, T.; McPhee, N.F.; Spector, L. Program synthesis using uniform mutation by addition and deletion. In Proceedings of the Genetic and Evolutionary Computation Conference, Kyoto, Japan, 15–19 July 2018; pp. 1127–1134. [Google Scholar]
- Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; Zaremba, W. Openai gym. arXiv
**2016**, arXiv:1606.01540. [Google Scholar] - Nielsen, M.A.; Chuang, I. Quantum computation and quantum information. Am. J. Phys.
**2002**, 70, 558. [Google Scholar] [CrossRef] [Green Version] - Kandala, A.; Mezzacapo, A.; Temme, K.; Takita, M.; Brink, M.; Chow, J.M.; Gambetta, J.M. Hardware-efficient variational quantum eigensolver for small molecules and quantum magnets. Nature
**2017**, 549, 242–246. [Google Scholar] [CrossRef] [Green Version] - Schuld, M.; Sweke, R.; Meyer, J.J. Effect of data encoding on the expressive power of variational quantum-machine-learning models. Phys. Rev. A
**2021**, 103, 032430. [Google Scholar] [CrossRef] - Spector, L. Automatic Quantum Computer Programming: A Genetic Programming Approach; Springer: New York, NY, USA, 2004; Volume 7. [Google Scholar]
- Grimsley, H.R.; Economou, S.E.; Barnes, E.; Mayhall, N.J. An adaptive variational algorithm for exact molecular simulations on a quantum computer. Nat. Commun.
**2019**, 10, 1–9. [Google Scholar] [CrossRef] [Green Version] - Tang, H.L.; Shkolnikov, V.; Barron, G.S.; Grimsley, H.R.; Mayhall, N.J.; Barnes, E.; Economou, S.E. Qubit-adapt-vqe: An adaptive algorithm for constructing hardware-efficient ansätze on a quantum processor. PRX Quantum
**2021**, 2, 020310. [Google Scholar] [CrossRef] - Las Heras, U.; Alvarez-Rodriguez, U.; Solano, E.; Sanz, M. Genetic algorithms for digital quantum simulations. Phys. Rev. Lett.
**2016**, 116, 230504. [Google Scholar] [CrossRef] [Green Version] - Du, Y.; Huang, T.; You, S.; Hsieh, M.H.; Tao, D. Quantum circuit architecture search: Error mitigation and trainability enhancement for variational quantum solvers. arXiv
**2020**, arXiv:2010.10217. [Google Scholar] - Lu, Z.; Shen, P.X.; Deng, D.L. Markovian quantum neuroevolution for machine learning. Phys. Rev. Appl.
**2021**, 16, 044039. [Google Scholar] [CrossRef] - Krastanov, S.; Albert, V.V.; Jiang, L. Optimized entanglement purification. Quantum
**2019**, 3, 123. [Google Scholar] [CrossRef] - Niu, M.Y.; Boixo, S.; Smelyanskiy, V.N.; Neven, H. Universal quantum control through deep reinforcement learning. NPJ Quantum Inf.
**2019**, 5, 1–8. [Google Scholar] [CrossRef] - Kuo, E.J.; Fang, Y.L.L.; Chen, S.Y.C. Quantum architecture search via deep reinforcement learning. arXiv
**2021**, arXiv:2104.07715. [Google Scholar] - Zhang, S.X.; Hsieh, C.Y.; Zhang, S.; Yao, H. Differentiable quantum architecture search. arXiv
**2020**, arXiv:2010.08561. [Google Scholar] [CrossRef] - Ostaszewski, M.; Grant, E.; Benedetti, M. Structure optimization for parameterized quantum circuits. Quantum
**2021**, 5, 391. [Google Scholar] [CrossRef] - Liu, Y.; Sun, Y.; Xue, B.; Zhang, M.; Yen, G.G.; Tan, K.C. A survey on evolutionary neural architecture search. IEEE Trans. Neural Netw. Learn. Syst.
**2021**. [Google Scholar] [CrossRef] - Ding, L.; Spector, L. Optimizing Neural Networks with Gradient Lexicase Selection. In Proceedings of the International Conference on Learning Representations, Vienna, Austria, 3–7 May 2021. [Google Scholar]
- Deb, K.; Agrawal, S.; Pratap, A.; Meyarivan, T. A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In International Conference on Parallel Problem Solving from Nature; Springer: Berlin/Heidelberg, Germany, 2000; pp. 849–858. [Google Scholar]
- Clerk, A.A.; Devoret, M.H.; Girvin, S.M.; Marquardt, F.; Schoelkopf, R.J. Introduction to quantum noise, measurement, and amplification. Rev. Mod. Phys.
**2010**, 82, 1155. [Google Scholar] [CrossRef] - Arute, F.; Arya, K.; Babbush, R.; Bacon, D.; Bardin, J.C.; Barends, R.; Biswas, R.; Boixo, S.; Brandao, F.G.; Buell, D.A.; et al. Quantum supremacy using a programmable superconducting processor. Nature
**2019**, 574, 505–510. [Google Scholar] [CrossRef] [Green Version] - Duan, Y.; Chen, X.; Houthooft, R.; Schulman, J.; Abbeel, P. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, New York, NY, USA, 19–24 June 2016; pp. 1329–1338. [Google Scholar]
- Hancock, A.; Garcia, A.; Shedenhelm, J.; Cowen, J.; Carey, C. Cirq: A Python Framework for Creating, Editing, and Invoking Quantum Circuits. 2019. Available online: https://github.com/quantumlib/Cirq (accessed on 15 November 2022).
- Broughton, M.; Verdon, G.; McCourt, T.; Martinez, A.J.; Yoo, J.H.; Isakov, S.V.; Massey, P.; Halavati, R.; Niu, M.Y.; Zlokapa, A.; et al. Tensorflow quantum: A software framework for quantum machine learning. arXiv
**2020**, arXiv:2003.02989. [Google Scholar] - Blank, J.; Deb, K. Pymoo: Multi-objective optimization in python. IEEE Access
**2020**, 8, 89497–89509. [Google Scholar] [CrossRef] - Williams, R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn.
**1992**, 8, 229–256. [Google Scholar] [CrossRef] - Greensmith, E.; Bartlett, P.L.; Baxter, J. Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning. J. Mach. Learn. Res.
**2004**, 5, 1471–1530. [Google Scholar]

**Figure 1.**Illustration of a simple 4-qubit architecture in the MEAS-PQC search space. The architecture (with genome encoding $1-2-3-0$) is composed of 4 operations: (1) Variational PQC (${\mathbf{x}}_{1}$) performs rotations on each qubits according to parameters $\theta $; (2) Entanglement (${\mathbf{x}}_{2}$) performs circular entanglement to all the qubits; (3) Data-encoding PQC (${\mathbf{x}}_{3}$) performs rotations on each qubit according to the input data d, scaling parameter $\mathsf{\lambda}$, and activation function $\sigma $; (4) Measurement (${\mathbf{x}}_{0}$) adds another Variational PQC (${\mathbf{x}}_{1}$) and performs measurement to obtain the observable values.

**Figure 2.**Learning performance on benchmark RL environments. We plot the learning curves (smoothed by a temporal window of 10 episodes) averaged over 20 trials of the resulting MEAS-PQC architecture compared against EQAS-PQC [15] and Softmax-PQC [11] on three benchmark RL environments. The shaded areas represent the standard deviation of the average collected reward.

**Figure 3.**Probability distribution of quantum operations over positions in top-performing PQC architectures. We select 30 top-performing architectures searched by MEAS-PQC (10 for each RL environment), and visualize the probability distributions of each operation over its positions.

Environment | # States/Qubits | # Actions | Reward | $\mathit{\gamma}$ | Horizon | Episodes |
---|---|---|---|---|---|---|

CartPole-v1 | 4 | 2 | $+1$ | $1.0$ | 500 | 600 |

MountainCar-v0 | 2 | 3 | $-1+height{\phantom{\rule{3.33333pt}{0ex}}}^{*}$ | $1.0$ | 200 | 1200 |

Acrobot-v1 | 6 | 3 | $-1$ | $1.0$ | 500 | 1200 |

Environment | Learning Rates (${\mathbf{\alpha}}_{\mathbf{\theta}}$, ${\mathbf{\alpha}}_{\mathit{w}}$, ${\mathbf{\alpha}}_{\mathit{\lambda}}$) | Observables | Value-Function Baselines | |||

CartPole-v1 | $0.01,0.1,0.1$ | [${Z}_{0}{Z}_{1}{Z}_{2}{Z}_{3}$] | None | |||

MountainCar-v0 | $0.01,0.1,0.01$ | [${Z}_{0},{Z}_{0}{Z}_{1},{Z}_{1}$] | Linear baseline [39] | |||

Acrobot-v1 | $0.01,0.1,0.01$ | [${Z}_{0},\cdots ,{Z}_{5}$] | Linear baseline [39] |

**Table 2.**Multi-objective Evaluation. For each method, we evaluate the performance based on the three objectives: learning performance, quantum noise, and computation cost, as described in Section 3.4. The results show that, with multi-objective optimization, MEAS can produce architectures with less quantum noise and computation cost, while still maintaining superior learning performance.

Environment | Objectives | Softmax-PQC [11] | EQAS-PQC [15] | MEAS-PQC (Ours) | ||
---|---|---|---|---|---|---|

1-obj | 2-obj | 3-obj | ||||

CartPole | Avg. Reward | 271.0 ± 78.8 | 317.6 ± 67.7 | 349.8 ± 71.3 | 351.6 ± 64.8 | 342.3 ± 76.2 |

Quantum noise | 30 | 20 | 25 | 17 | 15 | |

Model size | 20 | 12 | 19 | 13 | 13 | |

MountainCar | Avg. Reward | −126.6 ± 33.4 | −119.5 ± 32.8 | −108.2 ± 25.8 | −110.7 ± 21.4 | −111.3 ± 25.2 |

Quantum noise | 30 | 21 | 20 | 14 | 17 | |

Model size | 20 | 13 | 14 | 8 | 11 | |

Acrobot | Avg. Reward | −353.0 ± 93.7 | −328.5 ± 74.6 | −280.7 ± 79.3 | −283.4 ± 78.6 | −289.3 ± 75.5 |

Quantum noise | 30 | 17 | 22 | 19 | 15 | |

Model size | 20 | 13 | 16 | 11 | 11 |

**Table 3.**Pattern frequency in top-performing PQC architectures. We select 30 top-performing architectures searched by MEAS-PQC (10 for each RL environment), and calculate the frequency of patterns with different lengths. We can observe that consecutive ${\mathbf{x}}_{3}$ and alternating ${\mathbf{x}}_{1}$, ${\mathbf{x}}_{2}$ are the most common patterns.

len-2 | len-3 | len-4 | len-5 | ||||
---|---|---|---|---|---|---|---|

Pattern | Freq. | Pattern | Freq. | Pattern | Freq. | Pattern | Freq. |

(3, 3) | 0.282 | (3, 3, 3) | 0.213 | (3, 3, 3, 3) | 0.122 | (1, 2, 1, 2, 1) | 0.087 |

(2, 1) | 0.245 | (1, 2, 1) | 0.184 | (2, 1, 2, 1) | 0.122 | (1, 3, 3, 3, 3) | 0.075 |

(1, 2) | 0.205 | (2, 1, 2) | 0.142 | (1, 2, 1, 2) | 0.102 | (2, 1, 2, 1, 2) | 0.069 |

(1, 3) | 0.099 | (2, 1, 3) | 0.067 | (1, 3, 3, 3) | 0.068 | (3, 3, 3, 3, 2) | 0.064 |

(3, 2) | 0.066 | (3, 2, 1) | 0.059 | (3, 3, 3, 1) | 0.063 | (3, 3, 3, 3, 1) | 0.064 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ding, L.; Spector, L.
Multi-Objective Evolutionary Architecture Search for Parameterized Quantum Circuits. *Entropy* **2023**, *25*, 93.
https://doi.org/10.3390/e25010093

**AMA Style**

Ding L, Spector L.
Multi-Objective Evolutionary Architecture Search for Parameterized Quantum Circuits. *Entropy*. 2023; 25(1):93.
https://doi.org/10.3390/e25010093

**Chicago/Turabian Style**

Ding, Li, and Lee Spector.
2023. "Multi-Objective Evolutionary Architecture Search for Parameterized Quantum Circuits" *Entropy* 25, no. 1: 93.
https://doi.org/10.3390/e25010093