Pushing the Limits of Large Language Models in Quantum Operations

Abstract

What is the fastest Artificial Intelligence Large Language Model (AI LLM) for generating quantum operations? To answer this, we present the first benchmarking study comparing popular and publicly available AI models tasked with creating quantum gate designs. The Wolfram Mathematica framework was used to interface with the six AI LLMs, including Google Gemini 2.0 Flash, Anthropic Claude 3 Haiku, WolframLLM Notebook Assistant For Mathematica V14.3.0.0, OpenAI ChatGPT Omni 4 Mini, Google Gemma 3 4b 1t, and DeepSeek Chat V3. Our novel study found the following: (1) Gemini 2.0 Flash is overall the fastest AI LLM of the models tested in producing average quantum gate designs at 2.66101 s, factoring in the “thinking” execution time and ServiceConnect network latencies. (2) On average, four out of the ten quantum operations that the six LLMs produced compiled in Python version 3.13.5 (40.8% success rate). (3) Quantum operations averaged approximately 21–45 Lines of Code (omitting nonsensical outliers). (4) DeepSeek Chat V3 produced the shortest code with an average of 21.6 lines. This comparison evaluates the time taken by each AI LLM platform to generate quantum operations (including ServiceConnect networking times). These findings highlight a promising horizon where publicly available Large Language Models can become fast collaborators with quantum computers, enabling rapid quantum gate synthesis and paving the way for greater interoperability between two remarkable and cutting-edge technologies.

1. Introduction

The field of employing public Large Language Models in quantum gate operations is a novel topic of research. To our knowledge, there are no other results directly comparable to what we present in this paper. We conducted keyword searches on Google Scholar and found no matching results. However, we have added comparisons to other model papers where possible. The key difference between public and other models has to do with whether they are widely available general-purpose LLMs or not. This moniker stands in contrast to other “specialized LLMs,” such as those employed by Nicolas Dupuis et al. (more about these differences are discussed in the Literature Review section) [1]. Moreover, today’s quantum computers represent a crucial stage in an era of new performance and efficacy milestones. These systems, known as Noisy Intermediate-Scale Quantum (NISQ) computers, vary in size, “with a number of qubits ranging from 50 to a few hundred [2].” Key challenges confronting today’s NISQ systems include reliability and scalability.

Reliability-wise, NISQ systems face complex quantum noise environments. “Noisy emphasizes…imperfect control over those qubits; the noise will place serious limitations on what quantum devices can achieve in the near term [2].” When noise undesirably meshes with quantum information, that information becomes “entangled” with the environment in a process known as decoherence. Approaches to counter undesirable entanglement with the environment, or decoherence-induced errors in quantum computers, are addressed by the field of quantum error correction. This is a process whereby quantum information is encoded into a larger subspace of Hilbert space by monitoring symmetries of code space [3]. The end-goal is to achieve “fault-tolerance…the property that a circuit overall can be more reliable than the faulty gates that make it up [4].” This is especially relevant in “large-scale quantum computing [which] requires quantum error correction and fault-tolerance protocols to control the growth of errors [5].”

Scalability-wise, the hardware of NISQ systems faces challenges with control and confinement, or, again, coupling to the environment. This warrants approaches such as atomic, molecular, and optical physics and human-crafted hardware [3]. It should be mentioned here that some entanglements are desirable by design, as “the power of quantum parallelism [multiple operations] relies on the phenomenon [6].” Entanglement plays an “intriguing role…in quantum computing. As with most good things, it is best consumed in moderation [7].” While much research in the field focuses on the former two factors, we suggest a third factor may become a topic of consideration for quantum computers: interoperability, or the ability of quantum systems to effectively collaborate and communicate with other technologies and users. Historically, the design of quantum circuits has relied heavily on human expertise. However, the advent of accessible Artificial Intelligence Large Language Models (AI LLMs, or LLMs for short) and machine learning methodologies presents an opportunity to systematically explore and generate quantum circuit architectures that implement desired quantum operations. We argue this third factor of interoperability should be considered, especially “given the successes of both machine learning and quantum computing, combining these two strands of research is an obvious direction [8].”

Although research at the intersection of LLMs and quantum computing is still emerging, a crucial gap remains: a rigorous evaluation of publicly available AI models for quantum design applications. Our manuscript addresses this gap by delivering one of the first systematic benchmarks of leading AI models, such as Google Gemini 2.0 Flash, Anthropic Claude 3 Haiku, WolframLLM Notebook Assistant For Mathematica V14.3.0.0, OpenAI ChatGPT Omni 4 Mini, Google Gemma 3 4b 1t, and DeepSeek Chat V3, focused specifically on quantum circuit design and gate synthesis. In summary, such systematic comparisons of AI models specifically benchmarking fast quantum design and gate synthesis have been limited and nascent, until now.

2. Literature Review

The Web of Science is the gold-standard database of peer-reviewed journal articles. A title search within it for “quantum comput ∗” (where “∗” stands for a wild card; comput covers computation, computational, computer, computers, etc.) yields 8139 entries with the earliest published in 1975. The field of quantum computing or, more generally, quantum information science, is maturing rapidly with a number of research and review articles published in top journals. As examples of this, we provide the next three sections and perspectives.

2.1. Quantum Information Science Perspectives

The field of quantum information science is rapidly evolving. Ladd et al. (2010) in Nature explored whether storing, transmitting, and processing information encoded in uniquely quantum systems is feasible, noting that while promising, it remains unclear which technology will prevail [9]. O’Brien (2007) in Science highlighted that all-optical quantum computing became feasible with single-photon sources and detectors as early as 2001, though practical scaling presents challenges [10]. Belenchia, Wald, and Giacomini argued “that a quantum massive particle should be thought of as being entangled with its own Newtonian-like gravitational field, and thus that a Newtonian-like gravitational field can transmit quantum information [11].”

Wasielewski et al. (2020) in Nature Reviews Chemistry emphasized molecules’ quantum properties as new avenues for advancing quantum information science and its applications [12]. Moreover, this work has been further advanced by Moreno-Pineda and Wernsdorfer (2021), Toninelli et al. (2021), Harvey and Wasielewski (2021), Verma et al. (2026), De Bernardis et al. (2025), Alexeev et al. (2025), Morrillo et al. (2024), Hui et al. (2024), Mao et al. (2023), Chizzini et al. (2022), Scholes (2023), Chiesa et al. (2022), and over 400+ other works [13,14,15,16,17,18,19,20,21,22,23,24].

Bahrami et al. “computed the decoherence rate for two quite common types of environment: thermal radiation and background gas [25].” Additionally, Tóth and Apellaniz discussed applications for quantum Fisher information, such as ”how it can be used to obtain a criterion for a quantum state to be a macroscopic superposition [26].” Nayak et al. (2008), in Reviews of Modern Physics, focused on topological quantum computation using non-Abelian anyons, which offered fault-tolerance through nonlocal encoding [27].

Trapped-ion quantum computing, reviewed in Applied Physics Reviews by Bruzewicz et al. (2019), shows promise, with few-ion systems already demonstrating quantum algorithms [28]. Reiher et al. (2017) argued in PNAS that quantum devices are nearing computational power beyond classical supercomputers and are well suited for studying complex chemical reactions [29]. Raussendorf and Briegel (2001) proposed one-way quantum computers based on cluster states, where computation occurs via one-qubit measurements [30].

2.2. Quantum Error Correction Perspectives

Famously, John Preskill (2018) coined the name Noisy Intermediate-Scale Quantum (NISQ) devices, and in his paper, described the upcoming era of 50–100-qubit machines to outpace classical computers, though still fully expected fault-tolerant computing to require improved gates first to accomplish this feat [2]. Bourassa et al. (2021) advocated photonics as a modular, room-temperature platform for scalable, fault-tolerant quantum computing [31], while Steiger et al. (2018) introduced ProjectQ, an open-source framework to develop and simulate quantum algorithms [32].

Lohani et al. (2022) proposed data-centric machine learning methods that enhance quantum state reconstruction accuracy without changing model architectures [33]. Tóth and Apellaniz (2014) also reviewed quantum metrology advances and how noise limits precision [26]. Kitaev (2003) described fault-tolerant 2D quantum computation using anyonic excitations and their braiding operations [34]. We note that the “break-even point” of quantum error correction has so far only been demonstrated within the framework of the bosonic codes in superconducting and trapped-ion realizations (Long, 2024; Chizzini et al., 2022; Chlesa et al., 2022) [22,24,35].

2.3. Quantum and AI LLM Perspectives

We divide the Quantum and AI LLM Perspectives subsection into two further divisions, highlighting progress as well as the interoperability between these two technologies.

2.3.1. AI LLM Challenges and Innovations

Admittedly, with many new promising technologies, there are also expected challenges for LLMs, such as NP-hard (Nondeterministic Polynomial-time-hard) optimization problems. For example, this includes problems such as quantum Traveling Salesman Problems (TSP) [36]. These TSP solutions can be verified (in polynomial time), even if finding them might take an impractically long, non-polynomial time (like exponential time). However, because today’s LLMs lack “a truly robust reasoning mechanism,” or a logic module, we anticipate that LLMs will remain encumbered by NP-hard optimization problems for the foreseeable future [37].

Having said that, LLMs as a technology are also rapidly improving their capabilities, including coding accuracy. A number of benchmarks suggest that their capabilities are growing exponentially, with task complexity doubling roughly every 7 months. According to Bermejo et al. (2025), advanced LLMs now surpass even top human coders in accuracy for complex text analysis and select coding tasks [38]. Even though LLMs learn patterns from static data, while reinforcement learning (RL) methods learn behavior from dynamic feedback, we cautiously speculate that LLMs are likely to catch up with, and perhaps surpass, RL in the next few years. We also base this enthusiasm on the remarkable progress made between the publication of Sogabe et al. (2022) and Karuppasamy (2025) [39,40].

2.3.2. The Interoperability Between Quantum and AI LLM Technologies

In Current Opinion in Structural Biology, Lappala (2024) highlighted how integrating AI, machine learning, and quantum computing is revolutionizing molecular dynamics simulations, calling for multidisciplinary approaches to meet emerging challenges [41]. Moreover, other multidisciplinary approaches between LLMs and other fields are already emerging, such as those between LLMs and healthcare diagnostics [42,43,44]. And in their works, senior research scientists Srinivasan Arunachalam and Ronald de Wolf hinted at interoperability in several ways, including using “quantum machine learning for “quantum supremacy”, i.e., for solving some tasks  using 50–100 qubits in a way that is convincingly faster than possible on large classical computers [8].” In essence, could certain “practical machine learning problems [dovetail] with a large provable quantum speed-up? [8].”

At the same time, understanding the potential impacts of interoperability between technologies is significant, especially considering “the integration of artificial intelligence and quantum computing…could introduce new issues related to data quality [41].” Other researchers have also suggested interoperability between LLMs and quantum computers: work by Nicolas Dupuis et al. has discussed “an area of interest…to develop specialized LLMs for quantum code generation [1].” These specialized LLMs employed a “Granite code model…part of a family of decoder-based models for generative AI code tasks [1].” Furthermore, the authors recognized that “there is a noticeable gap in the application of machine learning and classical intelligence systems and algorithms to augment quantum ecosystems and platforms and empowering quantum computing practitioners [1].” In addition, Sanjay Vishwakarma et al. employed Qiskit HumanEval for “quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions [45].”

Although machine learning has been employed for generating/programming quantum gates [46,47,48], no comparison of efficiency of different public LLMs, which we consider here, has been offered. A similar story holds for neural networks [49,50,51,52] and for physics-informed neural networks [53]. Analogously, broadly relevant subjects, such as programmable quantum gate arrays [54], control of quantum gates [55,56,57], experimenting with quantum gates [58,59], scalable programmable quantum gates [60,61,62], operator-orderization of quantum gates [63,64,65], and others do not offer a comparison of the efficiency of different publicly available LLMs, which we consider here.

3. Materials and Methods

3.1. The Wolfram Mathematica Platform

The core of our Materials and Methods section involves using the Wolfram Mathematica framework. Significantly, we choose Wolfram Mathematica because it is a capable, trusted platform for research. Moreover, it has native integration with WolframLLM Notebook Assistant, and third-party support for a litany of other LLMs, including the 6 LLMs we tested, namely Google Gemini 2.0 Flash, Anthropic Claude 3 Haiku, WolframLLM Notebook Assistant For Mathematica V14.3.0.0, OpenAI ChatGPT Omni 4 Mini, Google Gemma 3 4b 1t, and DeepSeek Chat V3 (herein referenced by their single names: Gemini, Claude, WolframLLM, OpenAI, Gemma, and DeepSeek, respectively).

Wolfram Mathematica also includes a wide range of additional modules, such as those for quantum computing and machine learning [66,67]. Its underlying Wolfram Language, a symbolic programming language, empowers rapid prototyping to execution [68]. For example, tasks such as tallying numerical outputs or converting between units (such as between seconds and milliseconds) could be performed quickly. Combined with the framework’s extensions, modules, and in-house capabilities, this clearly drives our choice of Wolfram Mathematica for this study. In addition, by using a single framework, it ensures maximum effectiveness and consistent results.

3.2. The Methodology and Proposal

In real-world experiments, classical data is encoded into quantum states generally as follows in the diagram below, in a clock-wise order (top-left to bottom-right in Figure 1).

1.

Preparation/Code-Writing Stage: The quantum operations are input by a user into a networked computer, as shown in Figure 1.

2.

Networking/Transmission Stage: The operations are sent through the Internet via a framework such as Qiskit to a quantum center, like IBM’s Washington DC (us-east), as shown in Figure 1 and Figure 2.

3.

Pre-Measurement Stage: Through a hybrid classical/quantum control system, the request is passed to the actual quantum computer chip-set, e.g., the Heron r3 155-qubit system, as shown in Figure 2.

4.

Measurement/Execution Stage: The qubit executes the operations, the readout measures the results, and the data is returned back through the chain.

Figure 2. Snapshot from the IBM Quantum Experience Website.

Figure 2. Snapshot from the IBM Quantum Experience Website.

In our experiments, we focused our attention on the Preparation/Code-Writing Stage. The experiment’s first step here involved creating a master quantum prompt table (listed in the Appendix as

Table A5. Quantum prompts table part 1 (Prompts 1–5).

#	Concept	Prompt	Goal	Expected Output
1	Identity	Write code that builds a quantum circuit performing the identity operation. The code must begin with # START and end with # END. Return only executable code.	No change in state	Empty or canceling gates
2	Pauli X	Using only Hadamard and Z gates, write code that implements an X gate. The code must begin with # START and end with # END. Output the circuit.	Basis flip	H → Z → H
3	Pauli Y	Write code that constructs a Pauli-Y gate using only X and Z gates. The code must begin with # START and end with # END. Output only the quantum code.	Phase + bit flip	iXZ form
4	Pauli Z	Write code that expresses the Z gate using a sequence of T and T† gates. The code must begin with # START and end with # END. Return only executable code.	Construct Z from phase gates	T; T; T; T; or similar
5	Hadamard	Construct a Hadamard gate from rotation gates using code. The code must begin with # START and end with # END. Output only executable code.	X-basis projection	Rx/Rz decomposition

,

Table A6. Quantum prompts table part 2 (Prompts 6–10).

#	Concept	Prompt	Goal	Expected Output
6	CNOT decomposition	Write code that builds a CNOT gate using only CZ and Hadamard gates. The code must begin with # START and end with # END. Return the working circuit.	Functional CNOT	H; CZ; H
7	SWAP	Write a quantum circuit that swaps two qubits without using a SWAP gate. The code must begin with # START and end with # END. Output only executable code.	Qubit exchange	3 CNOTs
8	Toffoli (CCNOT)	Decompose the Toffoli gate using only H, T, and CNOT gates. The code must begin with # START and end with # END. Write executable code.	Control-control NOT	Standard decomposition
9	Fredkin (CSWAP)	Construct a Fredkin (CSWAP) gate using a Toffoli gate or other gates. The code must begin with # START and end with # END. Output only the code.	Controlled swap	Toffoli-based or ancilla-based
10	Bell state	Write code that prepares a Bell state between two qubits. The code must begin with # START and end with # END. Output the executable circuit only.	Create entanglement	H; CNOT

,

Table A7. Quantum prompts table part 3 (Prompts 11–15).

#	Concept	Prompt	Goal	Expected Output
11	Equal superposition	Create a 3-qubit quantum circuit that produces a uniform superposition. The code must begin with # START and end with # END. Return only code.	8-state	|$\psi$〉
12	GHZ state	Build a GHZ state circuit for 3 qubits using H and CNOT gates. The code must begin with # START and end with # END. Output only code.	Multi-qubit entanglement	H + CNOT + CNOT
13	W state	Write a circuit that prepares a W state using any method. The code must begin with # START and end with # END. Output only the code.	3-qubit, one-hot entanglement	Controlled superposition
14	Deutsch (no CNOT)	Write code that implements Deutsch’s algorithm without using the CNOT gate. The code must begin with # START and end with # END. Output executable code only.	Functional test of f(x)	H; CZ; H; or Oracle variant
15	Bernstein-Vazirani	Construct the Bernstein–Vazirani algorithm in code for the string s = 101. The code must begin with # START and end with # END. Return only the quantum circuit.	Reveal secret string	Hs + Oracle + H

,

Table A8. Quantum prompts table part 4 (Prompts 16–20).

#	Concept	Prompt	Goal	Expected Output
16	Grover’s 1-step	Write code for Grover’s algorithm on 2 qubits with a single marked item. The code must begin with # START and end with # END. Output executable code only.	Amplify marked amplitude	Oracle + diffuser
17	Simon’s algorithm	Implement Simon’s algorithm for a 2-bit function in executable code. The code must begin with # START and end with # END. Return only the quantum circuit.	Detect hidden XOR pattern	Register logic
18	Quantum Fourier Transform	Write a 3-qubit quantum Fourier transform circuit in code. The code must begin with # START and end with # END. Return only the code.	Phase encoding	Swap + Hadamard + CP gates
19	Logical AND (measured)	Construct a quantum circuit that behaves like a logical AND gate via measurement. The code must begin with # START and end with # END. Output only the code.	Output = 1 if both inputs = 1	CCNOT or entangled logic
20	Logical OR (measured)	Build a circuit that behaves like a logical OR gate using quantum measurement. The code must begin with # START and end with # END. Return executable code.	Output = 1 if any input = 1	Toffoli + ancilla or phase

and

Table A9. Quantum prompts table part 5 (Prompts 21–25).

#	Concept	Prompt	Goal	Expected Output
21	Entangle then swap	Entangle two qubits, then swap them. Write the code that does both. The code must begin with # START and end with # END. Output only code.	Confirm state transfer	Bell + SWAP
22	Quantum teleportation	Write a circuit that implements quantum teleportation for a single-qubit state. The code must begin with # START and end with # END. Output executable code.	Reconstruct qubit on receiver	Bell + CX + classical correction
23	Controlled-Z from CNOT	Construct a Controlled-Z gate using only CNOT and H gates. The code must begin with # START and end with # END. Write the code.	Control-Z logic	H; CNOT; H
24	XOR by measurement	Write code that encodes XOR into a quantum circuit and decodes by measurement. The code must begin with # START and end with # END. Output only the circuit.	0 if same, 1 if different	CNOT or parity circuit
25	Random entangled pair	Prepare a random Bell pair from an ancilla and Hadamard gates. The code must begin with # START and end with # END. Return only the executable circuit.	Bell state variation	Randomized entangler

), which listed every experiment that would be input. We proposed to have public Large Language Models devise the quantum gate operations as opposed to writing them ourselves, and to see how this would stack performance-wise between models. To do this, we devised four stages, outlined below.

First, the concepts chosen were an array of topics pulling from quantum gates (such as the Pauli Gate Set or Clifford Gates) to quantum algorithmic experiments (such as Quantum Fourier Transforms). Twenty-five prompts were elected and chosen to gain a sense of how LLMs would handle the subject materials. As an example, Appendix Table A5 lists a concept, a prompt for the LLMs we test, a goal of the prompt, and an expected output. In addition, the number of prompts was primarily chosen because of the limitations on how much execution time was permitted on each LLM service respective of cost and the project’s budget.

Second, we employed Wolfram Mathematica to input the prompts (found in Appendix Table A5, Table A6, Table A7, Table A8 and Table A9) in a notebook file, then one by one sent each of these prompts to an LLM service. During this time, we measured the “thinking” execution time, as well as the ServiceConnect network latencies from our computer running Wolfram Mathematica to the LLM services, and the time it took for the results to return back to us. Because it was not possible to measure the black box of the LLM itself (as the LLMs run on private third-party servers), these measurements allowed us to obtain a sense of how long each AI platform took to respond to prompt requests and connections. Unfortunately, at the time of writing this, measuring the connection time for the internal WolframLLM Notebook Assistant was not possible, as there was no native API support for measuring the connection time to this internal service.

Third, these results were then saved to text files, copied, filtered, and then indented so that they could be executed in their native language, such as Python. This entailed painstakingly and carefully removing the extraneous content from 150 prompts. This filtering included removing uncalled for header information or newline characters that the LLMs provided with every program. During the removal, we were careful to carve out only content specifically between the comments of #START and #END that we requested in the original prompts (available in the Appendix as Table A5, Table A6, Table A7, Table A8 and Table A9). When an AI model did not follow instructions, such as either providing a blank response or one that did not have appropriate termination, the prompt was left unedited.

Fourth and finally, the results were then executed in a Jupyter Notebook environment to see if they would compile. This environment was run on a Linux system with an Intel i7-7700HQ (3.80 GHz) with 16 gigabytes of RAM; see further specifications in Figure 3.

We looked at the success rates of each prompt measured by the output created and whether or not it would compile in a compiler. The below results are taken from Wolfram Mathematica, Jupyter Notebook, Python, and GenAI to produce open-source graphics from the corresponding data.

4. Results

To theoretically compare our performance results, we devised the following null and alternative hypotheses:

Hypothesis 0

(Null Hypothesis). There is no statistically significant difference in the average execution times among the LLMs when generating quantum gate operations from identical prompts.

Hypothesis 1

(Alternative Hypothesis). At least one LLM has a statistically different average execution time compared to the others.

We first graphed the results of all the LLMs, as shown in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9. The window chosen between 0 and 35 s was optimal for viewing the range of data from all six of the models. This range was also elected because of the generous feedback provided during this manuscript’s peer review process. Out of these models, five of the six LLMs generated consistently and relatively similarly low execution times for synthesizing quantum gates (Gemini, Claude, WolframLLM, Gemma, and OpenAI).

In contrast, DeepSeek was, on average, approximately twice as slow with its execution times. It is worth noting that since we do not have access to the black box of the LLM service providers, this execution time also included the time to file the request with the LLM service, as well as the “return trip” of the prompt. We will later discuss this trip to the service provider as a ServiceConnect time.

Gemini was the fastest LLM, on average, to execute our prompts.

As shown in

Table 1. Average LLM execution time per model (rounded to two decimal places, in seconds).

LLM	Average Execution Time (s)
Google Gemini 2.0 Flash	2.67
Anthropic Claude 3 Haiku	2.72
WolframLLM Notebook Assistant For Mathematica V14.3.0.0	3.48
Google Gemma 3 4b 1t	4.55
OpenAI ChatGPT Omni 4 Mini	5.22
DeepSeek Chat V3	12.23

, Gemini was followed by Claude, WolframLLM, OpenAI, Gemma, and DeepSeek, respectively. DeepSeek exhibited notably higher latency compared to the other LLMs.

Then, applying a significance test, the following were calculated:

Total time = 30.87 s;

Mean time ($\overline{x}$):

$$\overline{x}={\displaystyle \frac{30.87}{6}}=5.478 \mathrm{s}$$

Hypothesis 0

(Null Hypothesis). There is no statistically significant difference in the average execution times among the LLMs when generating quantum gate operations from identical prompts.

Hypothesis 1

(Alternative Hypothesis). At least one LLM has a statistically different average execution time compared to the others.

$$H_0:{\mu }_1={\mu }_2={\mu }_3={\mu }_4={\mu }_5={\mu }_6$$

$$H_a:\mathrm{At} \mathrm{least} \mathrm{one} {\mu }_i \mathrm{differs}.$$

Level of significance: $\alpha =0.05$.

5. Computation of Total Variance

$$SS=\sum {(x_i-\overline{x})}^2$$

$$\begin{array}{cc}\hfill SS& ={(2.67-5.478)}^2\hfill \\ \hfill & +{(2.72-5.478)}^2\hfill \\ \hfill & +{(3.48-5.478)}^2\hfill \\ \hfill & +{(4.55-5.478)}^2\hfill \\ \hfill & +{(5.22-5.478)}^2\hfill \\ \hfill & +{(12.23-5.478)}^2\hfill \\ \hfill SS& =65.952\hfill \end{array}$$

Degrees of freedom:

$$df=n-1=6-1=5$$

$$MS={\displaystyle \frac{SS}{df}}={\displaystyle \frac{65.952}{5}}=13.190$$

6. Approximate F-Test

We also perform an F-Test, which looks at approximate within-group variances of ${\sigma }^2=0.25$.

$$F={\displaystyle \frac{M{S}_{between}}{M{S}_{within}}}={\displaystyle \frac{13.190}{0.25}}=52.76$$

Critical value:

$$F_{0.05,(5,\infty )}\approx 2.21$$

Since $F=52.76\gg 2.21$, we have $p<0.001$.

Therefore, we reject $H_0$. There are statistically significant differences among the six models’ average execution times at $\alpha =0.05$.

7. Ranking

At the 95% significance level (with an $\alpha =0.05$), the mean execution times differ statistically significantly. In particular, DeepSeek shows a considerably longer average runtime, while Gemini performs the fastest overall, evident in

Table 2. Average execution times and rankings for each LLM (in seconds).

Model	Avg. Time (s)	Rank (1 = Fastest)
Google Gemini 2.0 Flash	2.67	1
Anthropic Claude 3 Haiku	2.72	2
WolframLLM	3.48	3
Google Gemma 3 4b 1t	4.55	4
OpenAI ChatGPT Omni 4 Mini	5.22	5
DeepSeek Chat V3	12.23	6

.

7.1. ServiceConnect Times

Next, ServiceConnect times were compared across five of the six LLMs tested (as an aside, the internal WolframLLM service does not support this API call within Wolfram Mathematica, so it could not be compared). It is worth reiterating that since we do not have access to the black box of the LLM service providers, we can only estimate the true execution time considering the time to file a request with the LLM service, as well as the “return trip” of the prompt. The initial trip to the service provider is known as the ServiceConnect time.

Also, the window chosen for our ServiceConnect tables was elected as 0–100 milliseconds, as this was optimal for viewing the range of data from all six of the models. Again, this was also chosen based upon peer review feedback in the making of this manuscript. These are composites of the total execution time measured above, but specifically the time that Wolfram Mathematica took to connect to each LLM service. We measured this to provide us with a picture of the networking latencies that could exist during testing. For example, using a different network could induce extra time in packet delivery from our system to that of the LLM providers.

Because of the volatility in these graphs, we plotted a five-point moving average of the data, as measured by the number of points (twenty-five points) and their respective times, averaged, as shown in

Table 3. Average ServiceConnect times for each LLM (rounded to two decimal places, in milliseconds).

LLM	Average Service Connect Time (milliseconds)
DeepSeek Chat V3	8.73
Google Gemini 2.0 Flash	8.99
OpenAI ChatGPT Omni 4 Mini	10.93
Anthropic Claude 3 Haiku	74.12
Google Gemma 3 4b 1t	76.10
WolframLLM Notebook Assistant For Mathematica V14.3.0.0	UNKNOWN

. Evidently, DeepSeek was the average relative fastest, followed by Gemini and OpenAI (Figure 10, Figure 11 and Figure 12). Gemma and Claude were the slowest (Figure 13 and Figure 14); however, they were tested at a different point in time in the writing of this work, so this likely explains the gap from the other LLMs, as seen in Figure 15. Additionally, we provided these data in tabular form to give another perspective on the results; see Table 3.

7.2. Quantitative Analysis

We sought to find out how common certain solutions were as LLM responses to our prompts. The major frameworks and languages in open-source quantum computing are Qiskit (Python), Cirq (Python), OpenQASM, Q# (Microsoft’s Quantum Development Kit), and Braket SDK (Software Development Kit, Python). We grouped these languages into four buckets: Qiskit, Cirq, OpenQASM, and Other, as shown in

Table 4. Quantum framework and language occurrences by LLM with total counts and proportional percentages.

LLM	Qiskit	Cirq	OpenQASM	Other	Total
DeepSeek Chat V3	22	0	1	2	25
Google Gemini 2.0 Flash	23	0	1	1	25
OpenAI ChatGPT Omni 4 Mini	25	0	0	0	25
Anthropic Claude 3 Haiku	22	0	0	3	25
Google Gemma 3 4b 1t	4	0	0	21	25
WolframLLM Notebook Assistant For Mathematica V14.3.0.0	23	0	0	2	25
Totals	119	0	2	29	150
Percent Overall (%)	79.3	0.0	1.3	19.3	100.0

. We were able to identify programming languages by keyword analysis (using regular expressions) as well as program statements belonging to a particular framework (e.g., Qiskit uses a Python import statement explicitly calling for Qiskit or a mention to QuantumCircuit, or Cirq is explicitly stated) [70,71]. When the prompt was empty, devoid of content, employed another framework, or something else occurred, we grouped this into the “Other” bucket. We then provided totals for this analysis.

Remarkably, as shown in Table 4, there was a considerable bias for Qiskit, clocking in at 79.3% of the provided solutions. In fact, even Gemini, a Google product, used Qiskit as opposed to Google’s native Cirq. Unexpectedly, Gemma employed a number of solutions not based on quantum frameworks at all, but in pure Python, C, and C++. OpenAI was the only LLM to purely use Qiskit.

7.3. Lines of Code (LoC) by Prompt

Next, we counted the Lines of Code (LoC) between the #START and #END comments that we had specified in our instructions, as seen in the master quantum prompt table (in the Appendix as Table A5, Table A6, Table A7, Table A8 and Table A9). Files that did not have #START and #END comment termination were estimated, and are marked with a single asterisk *, as shown in

Table 5. Lines of Code (LoC) per prompt across all LLMs, with average LoC per model.

Prompt	Claude	WolframLLM	DeepSeek	Gemini	Gemma	OpenAI
1	11	10	10	5	44 *	11
2	3	12	6	15	1	16
3	5	13	5	9	13 *	12
4	17	13	11	22	20	22
5	13	25	29	54	27 *	25
6	20	32	23	34	56	23
7	30	12	7	7	35 *	20
8	45	30	56	12	76 *	26
9	20	22	31	16	27 *	18
10	13	13	11	6	37 *	14
11	20	10	9	6	23	13
12	13	15	12	15	4093 **	14
13	28	39	23	16	24 *	38
14	56	50	51	57	44 *	47
15	33	24	28	39	45	41
16	29	39	39	32	44 *	31
17	46	40	38	53	35 *	35
18	32	32	14	25	30 *	22
19	35	36	32	27	37 *	34
20	39	34	37	51	34 *	45
21	14	23	20	20	25	25
22	43	36	31	71	50	37
23	26	17	16	17	49	18
24	33	33	20	13	48 *	33
25	16	14	14	19	4093 **	19
Average	24.9	25.9	21.6	26.3	341.9 **	25.7

* Files that do not have #START and #END comment termination are estimated. ** Gemma averages are heavily skewed by two very large outliers (both 4093 LoC). This is also reflected in the overall average of the column for Gemma. Without these, Gemma’s typical mean would be around the range of 38–45 LoC.

.

Average Six-Way Comparison

We then graphed the overall average Lines of Code per each prompt and per model, as shown in Figure 16. Because Gemma produced extreme outliers, we elected to use the upper range of our typical mean estimate of 45 LoC, as shown in the footnote of Table 5.

7.4. Execution Compilation Success Rate

Because the LLMs generally do not output Python or other programming language code in a compiler-friendly format, we often found that the code would not execute without the filtering process (discussed earlier in the Materials and Methods section). The compilation results and their respective failures and errors are the results even after this filtering. Some programs, such as OpenAI Prompt-2, required only new-line filtering, which was a consequence of the Large Language Model’s output among all the prompts and responses. Otherwise, OpenAI Prompt-2 produced output normally, as shown in Figure 17 and Figure 18.

We observed that OpenAI, Gemini, WolframLLM, DeepSeek, Gemma, and Claude performed at a 40.8% success rate (59.2% failure rate), as shown in Figure 19. This value was significantly impacted by several issues, including Gemma providing responses in a format not requested, as well as LLMs calling aer from qiskit, as opposed to the newer package, qiskit-aer. It is important to note that the successes reflected here only indicate that the program executed did not return an error.

As evident in

Table A1. LLM prompt errors and descriptions.

LLM	Prompt File	Error Description
DeepSeek	Prompt13_Output.txt	‘Classical bit index 0 is out-of-range.’
DeepSeek	Prompt18_Output.txt	name ‘np’ is not defined
Gemini	Prompt9_Output.txt	invalid syntax (<string>, line 1)
Gemini	Prompt22_Output.txt	‘Index 1 out of range for size 1.’
OpenAI	Prompt3_Output.txt	‘InstructionSet’ object has no attribute ‘num_…’
OpenAI	Prompt15_Output.txt	‘duplicate qubit arguments’
WolframLLM	Prompt13_Output.txt	‘QasmQobj’ object has no attribute ‘name’
WolframLLM	Prompt20_Output.txt	‘duplicate qubit arguments’

,

Table A2. LLM prompt errors and descriptions.

LLM	Prompt File	Error Description
DeepSeek	Prompt13_Output.txt	‘Classical bit index 0 is out-of-range.’
DeepSeek	Prompt18_Output.txt	name ‘np’ is not defined
Gemini	Prompt9_Output.txt	invalid syntax (<string>, line 1)
Gemini	Prompt22_Output.txt	‘Index 1 out of range for size 1.’
OpenAI	Prompt3_Output.txt	‘InstructionSet’ object has no attribute ‘num_…’
OpenAI	Prompt15_Output.txt	‘duplicate qubit arguments’
WolframLLM	Prompt13_Output.txt	‘QasmQobj’ object has no attribute ‘name’
WolframLLM	Prompt20_Output.txt	‘duplicate qubit arguments’

,

Table A3. LLM prompt errors and descriptions.

LLM	Prompt File	Error Description
Gemma	Prompt9_Output.txt	‘invalid syntax (<string>, line 1)’
Gemma	Prompt22_Output.txt	‘Index 1 out of range for size 1.’
Claude	Prompt1_Output.txt	‘QuantumCircuit’ object has no attribute ‘i’
Claude	Prompt15_Output.txt	‘cannot import name ‘Aer’ from ‘qiskit’ (/home/…’
Claude	Prompt11_Output.txt	‘cannot import name ‘Aer’ from ‘qiskit’ (/home/…’
Claude	Prompt22_Output.txt	‘cannot import name ‘execute’ from ‘qiskit’ (/h…’
Claude	Prompt13_Output.txt	‘cannot import name ‘Aer’ from ‘qiskit’ (/home/…’
Claude	Prompt16_Output.txt	‘cannot import name ‘Aer’ from ‘qiskit’ (/home/…’

and

Table A4. LLM prompt errors and descriptions.

LLM	Prompt File	Error Description
Claude	Prompt20_Output.txt	‘cannot import name ‘Aer’ from ‘qiskit’ (/home/…’
Gemma	Prompt24_Output.txt	‘cannot import name ‘execute’ from ‘qiskit’ (/h…’
Claude	Prompt19_Output.txt	‘cannot import name ‘Aer’ from ‘qiskit’ (/home/…’

, no prompt other than Prompt 13 was failed equally across LLMs. Even in this singular case of Prompt 13, the errors between LLMs (DeepSeek vs. WolframLLM) were completely different (note a QasmQobj error versus a classical bit index out-of-range error, as shown in Table A1).

8. Discussion

Following testing, it became clear from these results that Gemini was the fastest LLM at producing quantum operations. In all, if we consider the LLM execution time data from Table 1 (measured in seconds), which shows us all six of the LLMs’ “thinking” execution times, we see that Gemini was the fastest overall LLM to produce quantum operations, followed by Claude, WolframLLM, OpenAI, Gemma, and DeepSeek. Even if we account for ServiceConnect averages (which themselves had an overall minimal effect on overall total execution time, as shown in Table 3), this still does not change Gemini’s clear headway (considering Gemini’s 2.67 average execution time minus its 0.00899 ServiceConnect average time—which itself is included in the execution time—this yields 2.66101). Performing our test of significance of average execution times and ranks, we also see that Gemini performed the fastest overall, evident in Table 2.

However, based on the low success rate of the six LLMs, as shown in Figure 19, filtering of LLM output remains a key consideration to ensure executable code. We also noticed that because of the lack of #START and #END comment termination, Gemma suffered high levels of failure. Compilers expected a well-defined beginning and end to a program, which consequently brought down the overall success rate average of all of the LLMs tested. This is reflected well in Table 5. Based on programs that did terminate correctly, correct solutions were seen in prompts such as OpenAI Prompt-2. However, because of the amount of work required to filter and terminate a program correctly, it is our impression that the accuracy rate without factoring in these elements is likely very low compared to the expected results. Having said that, we provide the full results of all prompts for readers; see the Data Availability Statement below located in the Conclusion section.

Another important note that surprised us was that many of the LLM programmatic results relied upon the Qiskit framework, as opposed to others (all of this is shown in Table 4. While we cannot be (or perhaps better put, are not) allowed to see the “black boxes” of LLMs, these results suggest that the LLM training data may come from the same sources and therefore be biased toward certain quantum operations. A possible cause for this is simply that IBM Qiskit is that much more ubiquitous today in the open-source literature versus other sources, and henceforth, in the training data for LLMs. This contrast is stark when looking at Table 4. For instance, 79.3% of quantum operation prompt responses employed Qiskit, while none used Cirq across all six LLMs.

These are significant findings, as they demonstrate that current public LLMs are relatively fast, likely employ similar datasets, and can achieve positive success rates in producing functional quantum operations (if given the appropriate filtering). But how does this impact quantum gate synthesis in terms of the third key factor introduced earlier—interoperability, alongside reliability and scalability? We anticipate that as LLMs continue to improve, their applications in designing the quantum circuits for quantum computers of the future will expand accordingly.

Future research should focus on tracking the progression of gate synthesis performance in current and upcoming public LLMs, as well as analyzing historical improvements from earlier models. Crucially, the filtering capability of public LLMs should be evaluated further to see if they evolve at not just producing fast code, but by default, clean, filtered, and formatted code. In essence, we ask, will future public LLMs assess prompts and programs as requiring differently formatted output? Yet another exploration could investigate the accuracy of how certain quantum problems challenge or even thwart public LLMs (such as Grover Search problems versus Bell State problems). Also, future research could perform an inside look at the black box of public LLMs and how they create quantum solutions. It would be a limit of the imagination to list all the possibilities here; however, we can express this one thought for certain: the data presented here today suggests that publicly available LLMs employed for quantum gate synthesis remain an exciting avenue to watch!

9. Conclusions

We present the first benchmarking study comparing popular and publicly available Large Language Models tasked with creating quantum gate designs. The Wolfram Mathematica framework was used to interface with the six AI LLMs, including Google Gemini 2.0 Flash, Anthropic Claude 3 Haiku, WolframLLM Notebook Assistant For Mathematica V14.3.0.0, OpenAI ChatGPT Omni 4 Mini, Google Gemma 3 4b 1t, and DeepSeek Chat V3. Our novel study found the following:

• Gemini 2.0 Flash is overall the fastest AI LLM of the models tested in producing quantum gate designs, taking on average 2.66101 s, considering the “thinking” execution time and ServiceConnect network latencies.
• Across all six LLMs, roughly four out of ten generated quantum programs which compiled successfully (40.8% success rate).
• Quantum operations averaged between 21 and 45 Lines of Code (omitting nonsensical outliers).
• On average, DeepSeek produced the shortest code, averaging 21.6 lines.

These findings highlight a promising horizon where publicly available Large Language Models can become fast collaborators with quantum computers, enabling rapid quantum gate synthesis and paving the way for greater interoperability between two remarkable and cutting-edge technologies.