Bench of Euler: A Benchmark for Evaluating the Problem-Solving Abilities of Large Language Models

Abstract

Large language models (LLMs) have recently improved their problem-solving abilities and can solve complex mathematical problems with an increasing accuracy, necessitating the development of more challenging benchmarks. Over the years, the performance of LLMs on several benchmark datasets has also improved, motivating the development of evaluation frameworks that emphasize deeper algorithmic reasoning. This research presents Bench of Euler, a benchmark comprising 954 challenging mathematical and computational puzzles from the “Project Euler” collection, designed to evaluate complex multi-step mathematical reasoning. Solving these puzzles requires in-depth algorithmic deduction, as solutions based on brute force enumeration are mathematically infeasible. The benchmark is further classified into bands with respect to the problem difficulty to enable structured evaluation. Several existing open-source and proprietary, as well as base and reasoning, models have been evaluated on Bench of Euler, among which the highest-performing gpt-4o model by OpenAI achieved the highest accuracy of 43.21%. To evaluate the algorithmic efficiency, the models were prompted to generate Python code, which was subsequently executed in an isolated sandbox with strict timeout constraints. A failure analysis of the top-performing models revealed their inability to effectively optimize solutions, resulting in time-limit exceeded (TLE) errors during execution and outputs that often remained incomplete or approximate. Additionally, second-pass evaluations yielded an average performance improvement of approximately 2.09%, indicating limited gains from a second stochastic sample under the adopted decoding protocol.

1. Introduction

Large language models (LLMs) have recently improved substantially, progressing from simple prompt-based text generation to solving international Olympiad-level problems [1]. Once limited to syntactic fluency and basic comprehension tasks, LLMs have emerged as promising tools for structured problem solving [2], logical deduction [3], and mathematical reasoning [4]. This progress has been enabled by (a) increases in the training-data scale, (b) increases in the model size, and (c) advances in model architectures [5,6,7,8,9]. The performance of LLMs is further affected by the interaction among training-data composition, model scale, and fine-tuning strategies [10,11]. These led to the inception of state-of-the-art models, like openai/gpt-4o, anthropic/claude-opus-4-thinking, google/gemini-ultra-preview, etc., which have been reported to exhibit non-trivial problem-solving abilities in solving complex algorithmic and mathematical puzzles. Despite progress in surface-level generation, a fundamental limitation persists: LLMs often struggle to perform multi-step reasoning accurately and abstractly [12]. This difficulty is also reflected in multi-hop reasoning settings, where models must integrate information across multiple reasoning steps rather than rely on isolated local inferences [13]. Existing benchmarks such as GSM8K, MATH, and HARDMath, while valuable (

Table 1. Comparison between Bench of Euler and other related datasets, including the dataset size, the mode of sourcing, and the difficulty rating.

Authors, Year	Dataset	# Problems	Problem Sourcing	Difficulty
Hendrycks et al., 2021	Math [14]	12,500	Manual only	High School
Cobbe et al., 2021	Gsm8k [15]	8500	Manual only	Grade School
Fan et al., 2024	HARDMath [16]	1400	Algorithmic only	Graduate
Liu et al., 2024	MathBench-T [17]	632	Manual + Algorithmic	Undergraduate
Arora et al., 2023	JeeBench [18]	236	Manual only	High School
Frieder et al., 2024	Ghosts [19]	190	Manual only	Graduate
Sawada et al., 2023	ARB [20]	34	Manual only	Graduate
(Ours, 2025 +)	Bench of Euler	954	Algorithmic only	Graduate +

), are either limited to problems within specific difficulty ranges, which in most cases span undergraduate to graduate levels, or do not reflect the full rigor expected of an LLM intended to solve real-world mathematical problems.

Motivated by the community’s need for a benchmark with a manageable number of problems, but well-distributed difficulty levels and topics, this work proposes Bench of Euler, a dataset with 954 problems divided into bands (by difficulty) with problems derived from the Project Euler archive [21]. With a median difficulty rating of Graduate +, including problems that require research-level reasoning, Bench of Euler offers a blend of several topics, including (a) number theory, (b) combinatorics, (c) optimization, (d) algorithmic logic, etc. Furthermore, solving these puzzles requires mathematical intuition, algorithmic reasoning, and efficient coding, because brute-force solutions are often infeasible. Bench of Euler, when compared against 22 state-of-the-art LLMs (includes reasoning and base models and open and proprietary models) across both Pass@1 and Pass@2, openai/gpt-4o, achieved a peak accuracy of only 45.57%. While reasoning-augmented models achieved a higher average accuracy across the evaluated model pool ($+5.79$ percentage points in Pass@1 and $+6.06$ percentage points in Pass@2; Section 4), this trend does not hold uniformly at the individual-model level. The strongest model overall, openai/gpt-4o, a base model, outperformed every evaluated reasoning model, including o3-pro by 0.61 percentage points in Pass@1 and 0.76 percentage points in Pass@2, and claude-opus-4-thinking by 1.36 and 1.28 percentage points, respectively. The aggregate class-level difference should therefore be interpreted cautiously, as it is influenced by the composition and size of the base and reasoning model pools. Additionally, the failure analysis indicated that top-performing models may converge to suboptimal or approximate solutions [22,23], particularly from the second difficulty band.

Among other works that have used the Project Euler repository, PECC [24] combined Project Euler with Advent-of-Code problems (2396 total) to study prose-style requirement extraction in chat-based settings, but did not stratify Project Euler problems by difficulty nor did it isolate algorithmic-efficiency failure modes; its Euler subset also predates and is disjoint in scope from our band-wise, timeout-aware evaluation. MathArena’s Project Euler track [25] evaluated a small, weekly-growing set of the most recent (uncontaminated) problems using agentic, tool-augmented solvers with iterative code execution and answer submission—a complementary, but distinct, evaluation setting compared with our single-shot (Pass@1) and two-sample (Pass@2) direct-prompting protocol. Neither existing work provides (a) a large, difficulty-stratified four-band partition of Project Euler problems, (b) a 22-model comparison across the base vs. reasoning and open-source vs. proprietary axes, or (c) an explicit accounting of time-limit exceeded (TLE) failures as a first-class evaluation signal for algorithmic efficiency (Section 3.3). The Bench of Euler is therefore not a novel problem source, but a difficulty-banded subset (954 problems) together with the first large-scale, timeout-aware, base-vs.-reasoning systematic evaluation of 22 state-of-the-art LLMs.

Because solving Bench of Euler problems requires models to produce executable code, the benchmark is also comparable to established code-generation benchmarks. HumanEval [26] (164 hand-written function-completion problems) and MBPP [27] (974 entry-level problems) evaluate short, self-contained functions against unit tests, without mathematical modeling or algorithmic optimization. APPS [28] (10,000 problems) and CodeContests [29] scale to a competitive-programming-level algorithmic difficulty, but are graded only on test-case pass rates. LiveCodeBench [30] addresses contamination using time-windowed problem releases from coding contests, but, similar to APPS and CodeContests, it also does not require the multi-step mathematical derivation (e.g., number-theoretic or combinatorial insight) that Project Euler problems demand. SWE-Bench [31] targets an orthogonal capability—repository-level bug resolution in real-world codebases—rather than algorithmic problem solving from a natural-language specification. In contrast, the Bench of Euler requires models to (a) derive a mathematically correct algorithm (not only a syntactically valid program), (b) implement it efficiently enough to terminate within a strict wall-clock timeout (Section 3.3), and (c) produce an exact-matching numeric answer rather than pass a fixed unit-test suite—jointly evaluating mathematical reasoning and algorithmic efficiency in a way that neither pure math benchmarks (Table 1) nor pure code-generation benchmarks fully capture.

2. Problem Definition

This section defines the formal problem-solving setting targeted in Bench of Euler. Let $\mathcal{P}=\{{\mathcal{P}}_1,{\mathcal{P}}_2,\dots ,{\mathcal{P}}_N\}$ denote the set of mathematical problem instances ($N=954$ in this research), where each ${\mathcal{P}}_i$ is a natural language description of a computational problem that implicitly defines a total function $f_i:{\mathcal{I}}_i\to {\mathcal{O}}_i$, where ${\mathcal{I}}_i$ is the (nullary) input domain, as implied by the fixed structure of the problem, and ${\mathcal{O}}_i\subseteq \mathbb{Z}\cup \mathbb{R}$ is the answer space (discrete or real-valued, numeric). Each problem $P_i$ requires the subject to think of an algorithm ${\mathcal{A}}_i$ such that ${\mathcal{A}}_i\left({\mathcal{P}}_i\right)=o_i$, where $o_i\in {\mathcal{O}}_i$ is a unique, verifiable ground-truth output. Large language models are generative probabilistic models, ${\mathbf{M}}_{\Theta }:{\Sigma }^{\ast }\to \mathcal{D}(\mathbf{\Sigma })$ over the token space $\mathbf{\Sigma }$ [32], where $\mathcal{D}(\mathbf{\Sigma })$ is the distribution over next-token predictions. Problem solving is interpreted hereby as a sequence (token) generation process, conditioned on a prompt ${\mathbf{pr}}_i\in {\Sigma }^{\ast }$ derived from ${\mathcal{P}}_i$, resulting in $y_i={\mathbf{M}}_{\Theta }\left({\mathbf{pr}}_i\right)$ such that the final token, $y_i^{\mathrm{final}}$, is interpretable as an element of ${\mathcal{O}}_i$ and satisfies $y_i^{\mathrm{final}}=o_i$. LLMs follow an auto-regressive token generation, and thus, the decoding process is either (a) deterministic (e.g., greedy decoding or beam search [33,34] with temperature $T=0$) for Pass@1 or (b) stochastic decoding (e.g., top-p or nucleus sampling [35] with $T>0$) for Pass@k settings [26], where $k>1$ (here, $T=0.7$ for $k=2$). The correctness is defined by the indicator function as in Equation (1).

$${\mathbf{\Psi }}_i^{\left(k\right)}=\mathbb{I}\left[\exists j\in \{1,\dots ,k\} \mathrm{such} \mathrm{that} \mathtt{parse}\left(S_i^{\left(j\right)}\right)=o_i\right]$$

(1)

where, $S_i^{\left(1\right)},\dots ,S_i^{\left(k\right)}$ are k independently sampled sequences, and $\mathtt{parse}(·)$ is a normalization function. The overall Pass@k accuracy across the dataset is then defined as in Equation (2), a strict metric for exact answer matching.

$$\mathtt{Pass}@k={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\mathbf{\Psi }}_i^{\left(k\right)}$$

(2)

The complexity classes of these functions vary; while some are computable in $\mathsf{P}$, yet require algorithmic efficiency beyond brute-force due to large input bounds, others need $\mathsf{NP}$-hard combinatorial constructions where naive solutions are computationally infeasible within the model’s token budget. For example, computing ${\sum }_{n=1}^{{10}^9}\varphi \left(n\right)$ is theoretically tractable, but practically infeasible without an optimized algorithm. To obtain a final answer, the models generate Python code whose execution produces the numerical output. Although the LLMs do not execute code themselves, the generated programs are executed externally by the evaluation harness. Unlike other benchmark datasets, the Bench of Euler evaluates not merely the language fluency, but the LLM’s capacity to simulate algorithmic logic superficially.

3. Dataset

This section details the Bench of Euler dataset, consisting of 954 problems from nine different domains of mathematics—(a) number theory, (b) combinatorics, (c) geometry, (d) others, (e) logic, (f) arithmetic, (g) algebra, (h) optimization, and (i) probability.

Table 2. Topic-wise distribution of problems in the Bench of Euler dataset.

Topic	% of Problems	Description
Number Theory	37.20%	Focused on primes, modular arithmetic, factorization, etc.
Combinatorics	19.92%	Includes permutations, combinations, partitions, and counting.
Geometry	13.00%	Covers triangles, circles, lattice points, and spatial relationships.
Others	10.59%	Mixed or uncategorized problems that do not fit standard topics.
Logic	6.18%	Involves deductive reasoning, puzzles, and constraint satisfaction.
Arithmetic	5.87%	Covers operations on integers, digit manipulation, etc.
Algebra	2.94%	Involves equations, series, polynomials, and algebraic identities.
Optimization	2.83%	Focused on finding maximum or minimum under constraints.
Probability	1.47%	Based on expected value, randomness, and outcome distributions.

presents the topic-wise dataset distribution, the % composition, and the type descriptions for each topic under the Bench of Euler dataset. Unlike existing benchmark datasets, the Bench of Euler is composed of problems of varying difficulties, and therefore, it is divided into difficulty bands. Figure 1 further presents the band-wise topic distribution for the Bench of Euler dataset. Figure 2 presents a sample number-theory problem from the Bench of Euler, along with the chain-of-thought (solution) given by openai/gpt-4o. The following subsections will describe the dataset creation, the topic descriptions, and the corollary code evaluation judgment (internally through the dataset).

3.1. Dataset Construction

For building the Bench of Euler benchmark dataset, first, a set of 954 problems from the Project Euler archive were selected, scraped algorithmically and leaving behind no room for subjectivity bias [36] in the problem selection. To reduce the potential difficulty-selection bias [37], the puzzles were divided into bands based on the percentage difficulty level. It was observed that these metrics followed an inverted square law distribution. Problems with a normalized rating of less than 25% were set to be Band I (B1) of the dataset, ratings between 25% and 50% were considered Band II (B2) of the dataset, ratings between 50% and 75% were considered Band III (B3) of the dataset, and those problems that rated more than 75% were considered Band IV (B4), containing the toughest set of puzzles. The four difficulty bands contained 239, 239, 238, and 238 problems in B1, B2, B3, and B4, respectively. Thus, they formed near-equal partitions of the 954 problems. All problems in the dataset were stored using a structured JSON format with individual entries as:

[

    {

        "id": "1",

        "name": "Multiples of 3 or 5",

        "link": "https://projecteuler.net/problem=1",

        "html": "<p>...</p>",

        "Topic": "Arithmetic",

        "solvers": "285754",

        "pp_dif": "0",

        "solution": "233168"

    },

    ...

]

• The dataset includes fields for (a) the problem ID (aligned as Project Euler enumeration for brevity), (b) the standardized problem statement (in HTML format), (c) the correct answer, and the other difficulty metrics. The HTML encodings were further pre-processed to remove erroneous markup, normalize the mathematical expressions, and simplify phrasing for clarity and token efficiency when ingested by LLMs. The raw dataset (in JSON format) can be obtained from the corresponding author upon reasonable request. To augment this dataset with the LLM generated final answers, prompts were given to the LLM APIs with sleep(15) (being gentle to both the LLM and the wallet). For Pass@1, each LLM was provided with the problem statement and was expected to generate code in its first attempt without any retries or external hints. The prompt for Pass@1 was as follows,

• You are a mathematical problem solver. Read the problem carefully and write an optimized Python script to compute the exact numeric answer.

• Problem: [HTML Problem Statement]

• Answer:

• The generated Python code was further used for getting the numeric value, which was later compared to the ground truth for evaluation. For Pass@2, the same prompt was given to the model with same configuration, except that the decoding was allowed to be non-deterministic by enabling temperature-based sampling (temp = 0.7).

3.2. Topic Descriptions

The Bench of Euler benchmark includes puzzles from nine different topics (as mentioned previously). The dominant category is number theory, which comprises $\approx 37\%$ of the dataset, including problems such as prime generation, modular arithmetic, greatest common divisors, Euler’s totient function, integer factorization, etc. This category requires the use of sieve-based algorithms (optimized enough to be computed in finite time), modular inverses, Fermat’s theorems, and the application of other related advanced topics in higher mathematics. Combinatorics is the next dominant category, comprising ≈20% of the dataset. This category requires counting permutations, evaluating partitions, or exploring search spaces with combinatorial constraints using recurrence relations, memoization, or dynamic programming techniques. Geometry contributes to another $\approx 13\%$ of the dataset, and puzzles in this category require coordinate-based reasoning over Euclidean or lattice geometry, including the properties of polygons, circles, and triangle centers, and requires spatial reasoning, floating-point precision handling, and combinatorial geometry approaches. A significant subset falls under logic and constraint satisfaction, where the problem setup contains rules that must be satisfied simultaneously, solvable through backtracking, SAT solving, or simulation-based inference. Arithmetic problems make up approximately $6\%$ of the dataset; they often seem simple (by their appearance), but are computationally intensive, and they require large integer manipulation, digit patterns, or base conversions. Algebraic problems appear less frequently (about ≈3%) and require the symbolic manipulation of polynomial expressions, the evaluation of series, or the solving of equations under constraints. A small, but distinct, portion is also dedicated explicitly to optimization, wherein the objective is to minimize or maximize a quantity under algorithmic constraints; these problems require search-space pruning, greedy heuristics, or the use of graph-theoretic structures like spanning trees, shortest paths, etc. Probability-based problems, though minimal in count, require the estimation of expected values over discrete outcome spaces, requiring combinatoric enumeration or Markovian transitions. A remaining category of problems is grouped under a general others category. These problems have multiple overlapping domains and are often simulation-intensive, and those numericals do not fit well into traditional classifications. In the dataset, each problem is labeled with its domain category to enable domain-wise evaluation.

3.3. Code Evaluation Judgment

To evaluate the algorithmic reasoning and coding proficiency simultaneously, the models were prompted to generate a complete, executable Python script for each problem. The generated code was then extracted and executed within an automated evaluation harness deployed on a Google Colab (standard) CPU instance (Intel Xeon @ 2.20GHz with 12.7 GB RAM) running Python 3.10.

Additionally, to evaluate the code optimization, a strict spatiotemporal constraint was applied: each script was allotted a maximum execution wall-clock time of 60 s. Due to the scale of the evaluation, the testing was batched and executed iteratively (took nearly three weeks). If a script exceeded the 60 s. threshold, the runtime environment forcefully terminated the process and scored it as a time-limit exceeded (TLE) error [38]. Because the evaluation outcome is binary—i.e., the answer to a question can either be TRUE, if the final answer by the LLM matches with the solution, or FALSE otherwise—the problems with TLE errors also returned FALSE. Furthermore, the $\mathtt{parse}(·)$ function in Equation (1) is defined as capturing the standard output (stdout) of the executed Python script. The parsing pipeline strips formatting variants (e.g., trailing whitespaces, newlines, and thousands separators). Correctness is established through exact string matching of the extracted numerical output against the ground-truth solution. Figure 3 presents several code snippets written for solving puzzles from the Bench of Euler.

4. Experimentation

To justify the effectiveness of the Bench of Euler dataset, a comprehensive review of twenty-two state-of-the-art large language models was conducted, measuring both the Pass@1 and Pass@2 accuracies [26] under identical conditions.

Table 3. Accuracies for each of the 22 models (along with their providers) about both the passes.

Provider	Model Name	Reasons?	Pass@1 Accuracy	Pass@2 Accuracy
Anthropic	claude-opus-4-thinking	✓	0.4185	0.4429
	claude-3-7-sonnet-20250219	✗	0.3431	0.3691
	claude-sonnet-4-thinking	✓	0.3368	0.3573
	claude-sonnet-4	✗	0.3348	0.3426
	claude-3-5-sonnet-20241022	✗	0.3281	0.3487
	claude-3-5-haiku-20241022	✗	0.2756	0.2963
DeepSeek	deepseek-r1	✗	0.3165	0.3384
	deepseek-chat	✗	0.2832	0.3043
Google	gemini-ultra-preview	✗	0.2351	0.2789
	gemini-2.5-pro-preview	✓	0.1602	0.1777
	gemini-2.5-flash-preview	✗	0.1187	0.1341
	gemini-2.5-flash-lite-preview	✗	0.0908	0.1032
OpenAI	gpt-4o	✗	0.4321	0.4557
	o3-pro	✓	0.4260	0.4481
	o3-2025-04-16	✗	0.4043	0.4294
	o3	✗	0.4020	0.4226
	gpt-4.1-mini-2025-04-14	✗	0.3475	0.3691
	o3-mini	✗	0.3251	0.3453
	gpt-4.1-nano-2025-04-14	✗	0.2879	0.3087
xAI	grok-3-reasoner	✓	0.4005	0.4312
	grok-4	✗	0.2382	0.2447
	grok-3	✗	0.1754	0.1928

reports the Pass@1 and Pass@2 accuracies for all 22 evaluated models, together with their providers and model categories.

Each of the models was accessed via their documented APIs using the prompt (as mentioned in Section 3.1). For Pass@1, decoding was performed deterministically with a temperature of $T=0$ and greedy token selection; for Pass@2, each model was sampled twice with a temperature of $T=0.7$. An answer was counted as correct if, and only if, (a) the generated code successfully executed without syntax errors, (b) it completed within the strict timeout threshold, and (c) its parsed standard output exactly matched the ground truth. All API calls were rate-limited to one request per 15 s. to respect provider quotas, and executed over a three-week period (in total, 1908 queries (954 problems × 2 passes) were issued per model). Figure 4a,b show the Pass@1 and Pass@2 accuracies of the twenty-two large language models, where OpenAI’s gpt-4o (base variant) led with 43.21% and 45.57% overall correctness, followed by OpenAI’s reasoning variant o3-pro (42.60% and 44.81%), and claude-opus-4-thinking, Anthropic’s reasoning model (41.85% and 44.29%).

Aggregating by model class showed that the reasoning variants outperformed the base models by an average of 5.8 percentage points in Pass@1 and 6.1 points in Pass@2, and that proprietary models marginally exceeded the open-source models (refer to Figure 5). However, it was found that the aggregate advantage was not uniform across individual models on the Bench of Euler: the overall top performer, gpt-4o (base), outperformed every one of the five reasoning models evaluated on both passes—exceeding the closest reasoning competitor, o3-pro, by 0.61 pp (1.43% relative) in Pass@1 and 0.76 pp (1.70% relative) in Pass@2, and the weakest reasoning model, gemini-2.5-pro-preview, by 27.19 pp (169.73% relative) in Pass@1 and 27.80 pp (156.44% relative) in Pass@2.

Table 4. Failure-mode analysis (time-limit exceeded, logic error, and syntax error percentages) and average wall-clock execution time for each of the 22 models.

Provider	Model Name	TLE	Logical	Syntax	Wall Clock (s)
Anthropic	claude-opus-4-thinking	29.45%	27.51%	1.19%	25.39
	claude-3-7-sonnet-20250219	22.06%	37.61%	6.02%	21.40
	claude-sonnet-4-thinking	26.01%	36.75%	3.56%	25.87
	claude-sonnet-4	22.89%	37.61%	6.02%	21.81
	claude-3-5-sonnet-20241022	22.18%	38.72%	6.28%	21.45
	claude-3-5-haiku-20241022	15.49%	45.28%	11.67%	20.48
DeepSeek	deepseek-r1	23.19%	38.95%	6.21%	21.96
	deepseek-chat	15.59%	44.47%	11.62%	20.52
Google	gemini-ultra-preview	16.73%	48.02%	11.74%	21.10
	gemini-2.5-pro-preview	14.88%	49.74%	19.36%	24.37
	gemini-2.5-flash-preview	11.81%	52.67%	23.65%	20.21
	gemini-2.5-flash-lite-preview	12.16%	54.35%	24.41%	20.27
OpenAI	gpt-4o	26.06%	27.98%	2.75%	21.30
	o3-pro	29.04%	26.80%	1.56%	25.14
	o3-2025-04-16	27.65%	29.20%	2.73%	22.15
	o3	27.34%	29.16%	3.30%	21.95
	gpt-4.1-mini-2025-04-14	22.57%	37.25%	5.43%	21.71
	o3-mini	22.64%	38.86%	5.99%	21.71
	gpt-4.1-nano-2025-04-14	15.86%	44.02%	11.33%	20.67
xAI	grok-3-reasoner	30.37%	28.22%	1.36%	25.84
	grok-4	16.72%	47.04%	12.42%	20.98
	grok-3	10.44%	49.53%	22.48%	19.75

summarizes the time-limit exceeded, logical-error, and syntax-error rates, together with the average wall-clock execution time, for the 22 evaluated models.

Further, the band-wise accuracies (normalized by problem count, for consistency) for the top models under each provider were considered (refer to

Table 5. Provider-wise top models and their accuracies in the bands of the Bench of Euler dataset in Pass@1.

Provider	Model	B1 Accuracy	B2 Accuracy	B3 Accuracy	B4 Accuracy
OpenAI	gpt-4o	95.67%	72.10%	5.07%	0.00%
Anthropic	claude-opus-4-thinking	99.00%	61.74%	6.66%	0.00%
DeepSeek	deepseek-r1	77.10%	38.80%	10.70%	0.00%
xAI	grok-3-reasoner	94.10%	50.35%	15.75%	0.00%
Google	gemini-ultra-preview	84.08%	4.98%	4.98%	0.00%

and

Table 6. Provider-wise top models and their accuracies in the bands of the Bench of Euler dataset in Pass@2.

Provider	Model	B1 Accuracy	B2 Accuracy	B3 Accuracy	B4 Accuracy
OpenAI	gpt-4o	96.46%	79.13%	6.69%	0.00%
Anthropic	claude-opus-4-thinking	99.15%	67.99%	10.02%	0.00%
DeepSeek	deepseek-r1	79.07%	42.72%	13.57%	0.00%
xAI	grok-3-reasoner	97.35%	55.27%	19.86%	0.00%
Google	gemini-ultra-preview	91.73%	11.80%	8.03%	0.00%

) to analyze the performance across difficulty bands. It was observed that, in the first band (B1), gpt-4o (OpenAI, best performer) achieved 95.67% and 96.46% in Pass@1 and Pass@2, respectively, but upon escalating to the final (fourth band, B4), it was unable to solve even a single question. The same observation was evident across the remaining 21 models. Although all models scored 0.00% in B4, this represents a “floor effect” at the extreme difficulty level rather than an absence of discriminative power. We therefore used B3 to distinguish upper-end capabilities, and it showed that grok-3-reasonerwas the best performer on the most difficult non-saturated problems, with a 15.75% success in Pass@1 and a 19.86% success in Pass@2, compared to 4.98–15.75% and 6.69–19.86%, respectively, across the other provider-leading models. Additionally, to compare the relative performances (normalized) for all the LLMs from each of the providers (not limited to the best performers from each provider), pie-charts were constructed, as presented in Figure 6. It can be inferred from the plots that, particularly in B2 (the band with moderate–hard questions), the OpenAI models outperformed the competing models (for instance, with respect to its closest competition, xAI, OpenAI achieved 49.61% more merit) overall.

To evaluate the statistical robustness of the reported rankings (in Figure 4), we performed paired significance testing across all 22 models. Since the models were evaluated on an identical set of 954 problems, the per-problem correctness outcomes were naturally paired across models, and therefore, McNemar’s test [39] was applied to the paired Pass@1 correctness vectors for the top-scoring models to examine the discordant pairs (problems solved by one model, but not the other). The pairwise comparison between gpt-4o and o3-pro gives ${\chi }^2=4.50$, $p=0.0339$, and between gpt-4o and claude-opus-4-thinking gave ${\chi }^2=8.02$, $p=0.0046$ (both significant at $\alpha =0.05$), confirming that the observed accuracy ranking among the top models reflects genuine performance differences rather than sampling noise, despite the relatively narrow point-estimate gaps ($0.61$ and $1.36$ percentage points, respectively). We additionally note that Pass@2 was estimated from only $k=2$ stochastic samples per problem ($T=0.7$), smaller than the $k\ge 10$–100 typically used in the Pass@k estimator of [26] to reduce the sampling variance. Our choice of $k=2$ was primarily a cost and rate-limit constraint (Section 3.1: 1908 queries per model already required a three-week batched run under a 15 s. inter-request delay); increasing k would multiply the API cost and runtime roughly linearly. We recommend that future extensions of the Bench of Euler adopt $k\ge 10$ for Pass@k estimation to further stabilize the variance, though our paired significance tests indicate that the core Pass@1 ranking reported here is not an artifact of insufficient sampling.

Table 4 presents the failure rates for three common error types—(a) time-limit exceeded (TLE), (b) logical error, and (c) syntax error—with the average wall-clock execution time (in sec.) for each of the LLMs. It was found that a logical error was the most common error (averaging $39.53\%$ across the 22 models), followed by time-limit exceeded (averaging $20.96\%$), and then a syntax error (averaging $9.14\%$), the least frequent of the three. Common logical errors include (a) incomplete search-space pruning, (b) off-by-one indexing, and (c) approximate heuristic substitution for exact enumeration. In summary, the experimentations determined that (a) no existing LLM achieved greater than a 46% accuracy on the Bench of Euler, (b) reasoning-based decoding improves the class-average accuracy, and (c) the model performance degrades sharply with problem difficulty, thereby exposing the current gaps in algorithmic reasoning, and augmenting LLM development for the future.

5. Limitations

In this section, we identify some limitations of the present study that qualify the interpretation of our results.

5.1. Data Contamination

Project Euler has been publicly available since 2001, and solutions, discussions, and implementation guides for hundreds of its problems are extensively documented across the web, including GitHub repositories with thousands of solution implementations like nayuki/Project-Euler-solutions (≈10 years old) and igorvanloo/Project-Euler (≈4 years old), as well as the official Project Euler discussion forums, Stack Exchange threads, and personal blogs. Given that most state-of-the-art LLMs are trained on web-scale corpora, it is highly likely that a non-trivial fraction of Bench of Euler problems were present, in some form, in the training data of the evaluated models, particularly older, lower-numbered, and heavily-solved problems. This raises the possibility that the high accuracy observed on Band I problems (e.g., gpt-4o at $95.67\%$, Pass@1; Table 5) partially reflects the memorization of previously seen solutions rather than genuine algorithmic derivation. This contamination risk is not uniform across all difficulty bands. Band I problems are, by construction, the most frequently solved and most extensively discussed Project Euler problems (e.g., the “Multiples of 3 or 5” problem (PID:1) has been solved by hundreds of thousands of users and is one of the most commonly referenced introductory examples in online solution repositories, tutorials, and blog posts). Consequently, the B1 accuracy is the most susceptible to contamination-driven inflation. In contrast, B3 and B4 problems are, by construction, solved by far fewer users and correspondingly have substantially less publicly available discussion, solution code, or worked explanations online. The performance on B3 and B4 is therefore less likely to be inflated by memorization and more plausibly reflects genuine algorithmic reasoning under the models’ training-time knowledge. This asymmetry supports our choice of B3 (Section 4) as the primary band for comparing upper-end model capability.

A contamination study for future research would involve: (a) partitioning problems by publication date relative to each model’s stated training cutoff and comparing the accuracy on pre-cutoff versus post-cutoff subsets; (b) a perturbation analysis on a random problem subset, manually altering numeric constants while preserving the underlying algorithmic logic (e.g., replacing a stated bound of $\mathrm{12,000}$ with an arbitrary alternative), where a sharp accuracy drop under constant perturbation despite unchanged logical structure would indicate reliance on memorized answer values; and (c) a direct code-similarity comparison between model-generated solutions and known public implementations using similarity metrics.

5.2. Sampling and Statistical Limitations

As discussed in Section 4, Pass@2 was estimated from only $k=2$ stochastic samples per problem, substantially smaller than the $k\ge 10$–100 typically used in industry-standard Pass@k estimation [26] to reduce the sampling variance. With only two samples, the variance of the empirical Pass@2 estimate was considerably higher than that of a properly stabilized Pass@k estimate with $k\ge 10$, and small (<3 percentage point) differences between models under Pass@2 should not be treated as precisely resolved. This was a deliberate cost and rate-limit trade-off (Section 3.1). Accordingly, we clarify that Pass@2 in this study is reported as a supplementary reference point illustrating the direction (though not the precise magnitude) of gains from a second stochastic sample, rather than as a core evidentiary basis for our conclusions. Our core findings—the overall model ranking, the reasoning-versus-base comparison, and the band-wise discriminative power of the Bench of Euler—rely primarily on the deterministic, single-sample Pass@1 metric together with the paired McNemar significance tests reported in Section 4, which do not depend on the $k=2$ sampling budget. Future research could experiment with higher values of k to place Pass@k estimates on comparably rigorous statistical footing.

5.3. Failure-Mode Granularity

While Table 4 reports the TLE, logical error, and syntax error rates per model, this categorization reflects the final code generated by the LLMs, rather than an exhaustive, manual, problem-by-problem audit. The finer-grained sub-categorization of logical errors (e.g., distinguishing incomplete pruning from off-by-one indexing at the per-problem level) is left as a direction for future research. Additionally, the current dataset consisted of only 954 problems; this could be expanded over time, as new Project Euler problems are published regularly. The pool of 22 LLMs used in this research could also be expanded for a more comprehensive evaluation.

6. Conclusions

This work introduced the Bench of Euler, an algorithmically constructed, difficulty-banded benchmark dataset comprising 954 algorithmic and mathematical puzzles curated from the Project Euler corpus, divided into four difficulty bands and nine core domains of mathematics. The dataset construction pipeline, domain labeling, JSON schema formatting, ground-truth generation via cross-validated Python implementations, and prompt templates for both deterministic (Pass@1) and stochastic (Pass@2) decoding are detailed in this research. To evaluate the effectiveness of the Bench of Euler as a discriminative benchmark, a large-scale evaluation comparing twenty-two state-of-the-art LLMs was performed and reported in this research, which revealed that even the strongest model, gpt-4o, achieves less than a 50% accuracy, with the performance degrading sharply on higher-difficulty bands. Further, a comparison between open-source and proprietary and between base and reasoning-augmented models was conducted, alongside a systematic failure-mode analysis (Section 4) and an explicit discussion of the study’s limitations, including the data contamination risk (Section 5). Collectively, these findings establish the Bench of Euler not as a novel problem source, but as a curated, difficulty-stratified subset of Project Euler together with the first large-scale, timeout-aware, base-versus-reasoning systematic evaluation of 22 contemporary LLMs on it—one that remains a sustained challenge for present-day LLMs and decoding schemes, exposing critical gaps in algorithmic reasoning and efficiency.

Future research can extend the Bench of Euler through the integration of explicit code-execution feedback loops, hybrid models combining neural sequence generation with symbolic solvers, and curriculum-based pretraining on algorithmic reasoning tasks. Priority directions also include a formal contamination audit (Section 5.1), evaluation with a larger number of samples per problem ($k\ge 10$) for more stable Pass@k estimates, and the expansion of both the problem pool and the set of evaluated models. These extensions would make the benchmark more robust and sophisticated, and could potentially contribute to the development of the next generation of very large language models (VLLMs) toward better computational intelligence with mathematical reasoning.