Review Reports - Large-Scale Comparison of Metaheuristic Algorithms via Behavioral Verification of Large Language Model-Generated Codes

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1.The behavioral gates should not be treated as a ground truth measure of functional correctness.

The behavioral verification checks whether the generated code runs successfully, improves over the initial population, avoids pathological boundary behavior, preserves the best solution, and respects evaluation/output requirements on Sphere and Rastrigin. These checks are useful as implementation sanity tests, but they do not establish whether the code faithfully implements the intended published algorithm.The manuscript itself acknowledges that one gate-passing implementation later crashed on all CEC benchmark functions. More importantly, an implementation may pass all behavioral gates while using search operators or update rules that differ substantially from those defined in the original algorithm.I therefore suggest replacing terms such as “behavioral ground truth” and “functional correctness” with more precise terminology, such as “behavioral validity,” “execution-level validity,” or “sanity verification,” unless algorithmic fidelity is independently established.

2.The design of the LLM reviewer audit does not appear sufficient to evaluate whether LLMs can verify algorithmic correctness.

According to Section 3.2, the two LLM reviewers were given the algorithm name, generated code, and compile log. It is not stated that they were given the corresponding algorithm specification from the original publication.Without the original equations, pseudocode, or methodological description, an LLM reviewer cannot reliably determine whether the generated implementation is faithful to the intended algorithm. It can detect programming defects or suspicious code patterns, but it has no explicit reference against which algorithmic correctness can be judged.Therefore, the conclusion that LLM review provides essentially no information about functional correctness appears broader than what the current experiment supports. A more convincing design would provide reviewers with the original algorithm specification and explicitly ask them to assess consistency between the publication and the generated implementation.

3.The manuscript should explain how one implementation was selected from the three independent generation replicates for subsequent benchmarking.

The generation pipeline was repeated three times independently. Among the 393 algorithms, 250 passed in all three replicates, 111 passed only in some replicates, and 32 failed in all replicates. Nevertheless, the subsequent CEC benchmark uses one verified implementation for each of 361 algorithms. The manuscript does not clearly explain which implementation was selected when multiple successful implementations existed for the same algorithm.This selection rule is important because independently generated implementations of the same algorithm may differ significantly in behavior. The authors should explicitly state whether the first successful implementation, the last repaired implementation, a randomly selected implementation, or another criterion was used. Ideally, implementation-to-implementation variability should be analyzed for at least a representative subset of algorithms.

4.The statistical treatment of the LLM reviewer results should account for clustering and should avoid calling a non-significant result an equivalence result without a formal equivalence test.

The reviewer analysis pools 1179 generated drafts, but these correspond to three implementations for each of 393 algorithms. Therefore, the observations are not necessarily statistically independent at the algorithm level.An algorithm-level bootstrap, cluster-aware analysis, or mixed-effects model would provide a more appropriate estimate of uncertainty.In addition, the manuscript interprets p=0.73p=0.73 and A12=0.506A_{12}=0.506 as an “equivalence result.” A non-significant Mann–Whitney test does not by itself demonstrate statistical equivalence. If equivalence is to be claimed, a predefined equivalence margin and an appropriate equivalence testing procedure should be used. Otherwise, the result should simply be described as showing a negligible observed discrimination effect.

5.Applying Friedman–Nemenyi analysis only to the top-10 algorithms selected using the same data may introduce post-selection bias.

The manuscript first ranks all 360 algorithms using the CEC benchmark results and then applies the Friedman test and Nemenyi post-hoc procedure only to the top-10 algorithms identified from those same results. Because the subset was selected based on the same data used for statistical inference, the reported significance differences may be affected by post-selection bias.The authors should either treat the top-10 analysis as exploratory, apply statistical inference to a predefined comparison set, or use a resampling procedure that repeats both ranking and top-10 selection within each resample.

6.The performance conclusions should consistently distinguish between algorithms and specific LLM-generated implementations.

The manuscript itself shows that 49 of 50 LLM-generated implementations were statistically divergent from reference-library implementations of the same named algorithms. This is an important result because it demonstrates that algorithm identity cannot be assumed solely from the algorithm name.At the same time, the benchmark section sometimes makes broad statements about one algorithm outperforming another. Since the hyperparameters were chosen by the LLM and the implementations were generated independently rather than necessarily following official reference code, these conclusions should be formulated more carefully.

Author Response

Please see the attached point-by-point response letter.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This paper develops an execution-based generation and repair pipeline for 393 metaheuristic algorithms, audits two LLM code reviewers against behavioral checks, compares 50 generated implementations with a reference library, and benchmarks 360 implementations on the opfunu versions of CEC 2017 and CEC 2022 at ten dimensions. The reported improvement in gate-passing rates, frequent performance divergence between implementations, and strong rankings of Differential Evolution provide useful evidence about the practical limits of LLM-generated optimization code. The scale and released artifacts are valuable contributions. I recommend minor revision, with the findings framed as results for the evaluated implementations and testing conditions and the following clarifications addressed.

1. Please clarify whether generation received only algorithm names and the interface, as Section 3.1 states, or also the original publications and equations. Please provide the corpus selection source, publication-to-implementation mapping, prompt template, and model decoding settings. If publications were not supplied, please avoid attributing generation failures directly to gaps in their specifications.

2. Please give one consistent specification of G0–G4, including the improvement gate G1, which is absent from the methods list. Please reconcile whether G2 examines the final population or best solution and whether G4 checks seeding or budget/finiteness. Please also define the G3 tolerance and how all six function–seed tests determine acceptance.

3. Please reconcile the pooled 78.4% success rate in Table 1, the repair percentages reaching 94.7% in Section 4.1, and the union of 361 successful algorithms across replicates. Please state each denominator and explain which passing implementation was selected for downstream experiments when multiple replicas succeeded, including whether selection used performance information.

4. Please define the reviewer approval rules precisely: Table 3's single-reviewer column appears to mean at least one approval, while a mean score of at least 4.0 does not necessarily mean both reviewers individually approved. Please report reviewer-specific results, score-parsing validation, and uncertainty accounting for three drafts per algorithm, and qualify the claim of no information at any threshold to the analysis actually performed.

5. Please explain how G4's budget requirement is compatible with 31% of fidelity runs requiring external cutoff. Please document initialization counting, interruption handling, the returned best-so-far solution, and checkpoint recording, and distinguish a common evaluation cap from equal evaluations actually consumed, particularly for the under-consuming Pelican implementation.

6. I would suggest describing Grey Wolf Optimizer as showing no detected divergence under the stated tests unless an explicit equivalence analysis supports a stronger conclusion. Please specify the Holm correction family and avoid treating non-significance as proof of equivalence. Please also separate effects of different defaults from verified differences in update mechanisms when interpreting the 49 divergent pairs.

7. Please distinguish the global ranks over 360 implementations from the re-ranked, selected top-ten comparisons in Figure 4. Please explain how selecting contenders on the same benchmark affects inferential claims and describe the function-bootstrap intervals as conditional on the evaluated suite. The non-significant CEC 2022 omnibus result should accompany claims of superiority, alongside the ten-dimensional and modified-suite scope.

8. Please give the exact performance-ratio formula for Figure 5, including whether its denominator uses the best of ten or all 360 implementations, and explain sensitivity to the 10^-8 error offset and ties. Please describe these final-budget profiles as robustness across functions; claims about robustness across evaluation budgets would require using the recorded intermediate checkpoints.

Author Response

Please see the attached point-by-point response letter.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The revised manuscript has been substantially improved.I have only a few remaining comments:
The abstract states that 32 algorithms failed identically across replicates, whereas Table 2 indicates that only 24 exhibited the same failure signatures. This distinction should be clarified. The description of G2-related failures in Table 2 should also be checked for consistency.

Author Response

Response:

I thank the reviewer for the positive feedback. The 32 algorithms failed in all three replicates, and the 24 are those that failed with the same failure signature in all replicates. This is now clarified in the abstract, Section 4.1, Table 2, and the Conclusion. The G2 descriptions in Table 2 were updated to state "G2 (pinned against the search range)" and "G1 + G2 (no improvement; pinned against the search range)", and the related sentence in Section 4.1 was aligned with the corrected G2 definition.