Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix Benchmark

Abstract

Benchmark results for large language models often show inconsistencies across different studies. This paper investigates the challenges of reproducing these results in automatic bugfixing using LLMs, on the HumanEvalFix benchmark. To determine the cause of the differing results in the literature, we attempted to reproduce a subset of them by evaluating 12 models in the DeepSeekCoder, CodeGemma, CodeLlama, and WizardCoder model families, in different sizes and tunings. A total of 35 unique results were reported for these models across studies, of which we successfully reproduced 12. We identified several relevant factors that influenced the results. The base models can be confused with their instruction-tuned variants, making their results better than expected. Incorrect prompt templates or generation length can decrease benchmark performance, as well as using 4-bit quantization. Using sampling instead of greedy decoding can increase the variance, especially with higher temperature values. We found that precision and 8-bit quantization have less influence on benchmark results.

1. Introduction

Large language models (LLMs) have demonstrated remarkable capabilities in software development [1,2,3], which include their ability to detect and resolve bugs. Various benchmarks are designed to evaluate these capabilities, such as QuixBugs [4], HumanEvalFix [1], BugsInPy [5], and MDEVAL [6].

The most widely acknowledged bugfix benchmark is probably HumanEvalFix, a component of the HumanEvalPack [1]. In this benchmark, each of the 164 canonical solutions from the original HumanEval code generation benchmark [7] has been manually corrupted by introducing a bug. The task of the LLM is to repair the buggy function. The modified output is then verified using the original test cases.

Although reliable reports of benchmark results are essential to compare models, it is not uncommon to see differing benchmark results for the same model throughout studies [8,9,10,11,12]. In the broader community, researchers can struggle to reproduce the officially reported benchmark results. The difference between the reported and reproduced results can be minor, but it can also be significant. This not only complicates the comparison of models but also raises questions about the validity of the currently published benchmark results.

In this paper, we address the issue of inconsistent benchmark results on the HumanEvalFix benchmark. We look for discrepancies in a broad range of results published in the literature, considering models with scores reported in two or more studies. To identify the underlying reasons for these discrepancies, we conduct our own evaluations using 12 models of various sizes and tunings: DeepSeekCoder (1.3B and 6.7B, base and instruct) [3], CodeLlama (7B and 13B, base and instruct) [2], CodeGemma (2B base, 7B base and instruct) [13], and WizardCoder (15B) [14]. Our findings are as follows:

• We reproduce 16 of the 32 scores reported for the evaluated models. This implies that many of the discrepancies in the reported results can be attributed to using different or possibly incorrect evaluation settings.
• We quantify the impact of modifying the evaluation settings individually. For instance, the maximum generation length, sampling strategy, 4-bit quantization, and prompt template choice significantly influence the results, whereas precision and 8-bit quantization do not.
• We identify instances of possibly misreported results, likely due to confusion between base and instruction-tuned models.

2. Related Work

Some studies have raised questions about the reproducibility and reliability of benchmark results of LLMs, identifying sources of variance that can undermine them. A clear consensus in the field is that even minor differences in evaluation methods and settings might lead to significant discrepancies.

Biderman et al. investigated the impact of different prompts on model performance, demonstrating their significant influence on benchmark results [9]. To address the reproducibility issues, they propose a checklist for reporting benchmark results. This checklist suggests disclosing the evaluation setup and sharing the exact prompts, the evaluation code, and the model outputs. Furthermore, they recommend against copying results from other papers without verification. They propose the Language Model Evaluation Harness framework as a tool to facilitate greater reproducibility.

Laskar et al. point to the lack of transparent evaluation details [10]. They highlight the issue of not disclosing the precise model versions, which can lead to inconsistencies. The authors also address potential problems originating from the prompt design, including the practice of “prompt hacking”, manipulating input prompts to elicit desired responses. They also emphasize the lack of transparency concerning decoding parameters, such as the temperature, and note issues related to excessive limitations on generation length.

Hochlehnert et al. studied the heightened sensitivity of reasoning models to decoding parameters [11]. They propose a standardized framework for model evaluation with clearly defined best practices and reporting standards. Applying this framework revealed decreased performance for several models, suggesting that their previously reported results might have been influenced by optimized evaluation parameters. Their work also investigated the impact of random seeds, showing that single-seed evaluations can be unstable. They averaged results over multiple seeds to obtain a more accurate representation of performance. They observed that higher temperature values can increase the accuracy but also lead to greater variability, while higher top_p values improve the performance without compromising the stability. Other factors contributing to performance variance are also discussed, such as differences in hardware settings and software frameworks.

Yuan et al. emphasized the crucial role of numerical precision in reproducible LLM evaluation [12]. They recommend using fp32 precision to reduce rounding errors. They also measured the extent to which factors like the GPU count, GPU hardware version, and evaluation batch size influence benchmark results. They find that bf16 precision is sensitive to variations in hardware and system configurations. To mitigate this, they propose an optimized inference pipeline called LayerCast, which stores model weights in 16-bit precision but performs all computations in fp32, thereby balancing memory efficiency with numerical stability.

3. Method

We begin by introducing the benchmark we focus on, HumanEvalFix, as well as the evaluation framework commonly used to run it. Then, we identify the discrepancies across the reported results by conducting a literature review of the benchmark scores. Finally, we analyze GitHub issues on the unsuccessful reproduction of published results to determine possible reasons for these discrepancies and our experimental setup.

3.1. Benchmarking with HumanEvalFix

The authors of the HumanEvalFix benchmark have introduced a bug into each of the 164 canonical solutions in the HumanEval benchmark, thereby turning it into a bugfixing benchmark. These bugs involve missing logic, extra logic, or incorrect logic. The benchmarked LLM is prompted with the incorrect function, along with unit tests, and is tasked with repairing the function. The output code is then validated using the same unit tests found in the original HumanEval benchmark. Since the Python-based variant of this benchmark is the most widely used, we also concentrate our efforts on it.

For conducting our evaluations, we use the Code Generation LM Evaluation Harness [15] framework, provided by the BigCode Project. This framework includes multiple benchmark implementations, such as HumanEval, MBPP, and QuixBugs, as well as the HumanEvalFix benchmark. Most of these benchmarks evaluate the functional correctness of the code outputs using the pass@k metric, which indicates the percentage of outputs that pass the tests at least once in k attempts. Researchers often use a larger number of samples (>k) to provide a more refined pass@k estimate. Pass@1 is often reported, which measures whether the first generated output passes all test cases. Users of the framework can adjust a range of evaluation settings, such as the maximum generation length, batch size, and sampling parameters. The maximum generation length controls how many tokens the model can produce in a single output (including its prompt tokens). The batch size determines how many inputs are processed simultaneously. The precision settings (fp16, bf16 and fp32) determine how numbers are represented during computation. Quantization reduces the model size by compressing the numerical representations, often with minimal performance loss. Sampling parameters include the temperature, which affects the randomness in output. Top-p restricts generation to the most probable tokens whose cumulative probability meets a threshold, helping control diversity.

3.2. Review of Reported Benchmark Results

We conducted a review of 17 papers that contained results of the HumanEvalFix benchmark [1,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30], focusing on their reported HumanEvalFix results. These papers reported benchmark results for a total of 117 different models (including variants in sizes and tunings). Out of the 117 models mentioned, 25 appeared in at least two papers. These models formed the basis of our investigation. Their reported results are visualized in Figure 1, as well as in

Table 1. All reported HumanEvalFix results of LLMs considered in this paper. Citations are provided for studies introducing models and for ones that evaluate them.

Model	Reported Results
DeepSeekCoder-Inst (33B) [3]	47.5 [16,22], 81.0 [27]
DeepSeekCoder-Inst (6.7B) [3]	42.1 [21], 44.9 [16,22], 56.1 [30], 60.4 [17], 73.3 [27]
DeepSeekCoder-Inst (1.3B) [3]	9.1 [16], 29.3 [21], 48.9 [27]
DeepSeekCoder (6.7B) [3]	23.8 [21], 29.9 [30], 45.4 [27]
DeepSeekCoder (1.3B) [3]	1.2 [21], 16.4 [27]
CodeGemma-Inst (7B) [13]	46.3 [23], 72.4 [27]
CodeGemma (7B) [13]	8.5 [23], 11.7 [27], 53.7 [17]
CodeGemma (2B) [13]	4.3 [23], 22.7 [27]
CodeLlama-Inst (34B) [2]	36.5 [16,22], 37.8 [23], 55.9 [27]
CodeLlama-Inst (13B) [2]	15.2 [20], 16.4 [24], 18.9 [23], 19.4 [22], 29.2 [30], 30.5 [27]
CodeLlama-Inst (7B) [2]	15.2 [24], 19.5 [23], 20.6 [27], 28.0 [30]
CodeLlama (34B) [2]	14.0 [23], 47.6 [27]
CodeLlama (13B) [2]	6.1 [23], 33.1 [27]
WizardCoder (15B) [14]	31.8 [1,20,24,30], 51.6 [27]
Llama3-Inst (70B) [31]	57.3 [23], 81.7 [27]
Llama3-Inst (8B) [31]	40.2 [23], 41.5 [27]
OctoCoder (16B) [1]	28 [23], 30.4 [1,20,22,24,25,30]
OctoGeeX (6B) [1]	28.1 [1,24]
BLOOMZ (176B) [32]	16.6 [1,20,25]
StarCoder2 (15B) [22]	9.1 [23], 9.7 [22,25]
StarCoderBase (15B) [33]	10.4 [23], 12.6 [22,25]
StarCoder (16B) [33]	8.7 [1,20,24,30]
CodeGeeX2 (6B) [34]	15.9 [1,20,24]
InstructCodeT5+ (16B) [35]	2.7 [1,20,24]

, with all relevant citations.

Six out of 25 models have identical reported results across studies (e.g., InstructCodeT5+, StarCoder). This suggests that the initially published result was cited by other authors without trying to reproduce it. For 4/25 models, the results differed only slightly (e.g., OctoCoder, Llama3-Inst-8B). Small differences are accepted in many cases as they can be due to minor differences in configuration settings. However for most of the models, more precisely in 15/25 cases, the results differed significantly (e.g., CodeLlama, DeepSeekCoder). For example, the reported results for DeepSeekCoder-Instruct (1.3B) ranged from $9.1\%$ to $48.9\%$. Similarly, for CodeLlama-Instruct (13B), six benchmark results are reported, ranging from $15.2\%$ to $30.5\%$, nearly evenly spread. According to one reported result, DeepSeekCoder-Instruct (33B) should be one of the top models for bugfixing with an exceptionally high score of $81.0\%$; however, two other studies report a much lower score of $47.5\%$.

A factor that complicates cross-study comparisons is that models are sometimes referenced with slightly different parameter counts. For example, two papers list a specific version of WizardCoder as a 15B model [14,30], another as 15.5B [20], two others as 16B [1,24], and another as both 15B and 15.5B [27]. Although these differences probably originate from varied rounding/flooring, they nonetheless hinder model identification. In our work, we treat this version of WizardCoder as a 15B parameter model.

3.3. Experimental Setup

The goal of our experiments was twofold. First, we examined the effect of a wide range of settings on the results of models mentioned in multiple studies. Then, we tried to reproduce the results reported in the literature by determining the evaluation settings they could have used. In this section, we describe the experimental setup for the evaluations.

To explore potential causes for the differences in the benchmark results, we reviewed the reported issues about unsuccessful attempts at reproducing published results in the GitHub repository of the benchmarking framework (https://github.com/bigcode-project/bigcode-evaluation-harness). At the time of investigation, 145 issues were published (47 open and 98 closed). We reviewed these issues on the 5th of November 2024. Thirteen out of the 145 issues (six open and seven closed) were about the unsuccessful reproduction of officially reported results. We analyzed the issues that were either closed or had helpful answers to them (10 in total), and found the following causes (links to these issues are provided in the Appendix A.2):

• A differently tuned version of the model was used (e.g., the base model instead of its instruction-tuned variant), causing a large difference in benchmark result. (Two issues)
• The wrong variant of the same benchmark was used: MBPP instead of MBPP+ and HumanEval instead of its unstripped variant. (Two issues)
• The temperature was set incorrectly: while generating one sample for each prompt, the temperature $T=0.8$ turned out to be an improperly large value. (One issue)
• An incorrect prompt was used, causing more than $6\%$ of a decrease in pass@1 performance. (One issue)
• The reproduced results were different by only a few percentage points compared to the officially reported ones. Such a discrepancy was considered negligible, as this can be due to minor variations in evaluation settings, minor updates in model versions, or in hardware configurations. (Three issues)

Most papers provide details about their evaluation setups, but these can sometimes be insufficient for full reproducibility. For instance, the precision used during evaluation is never specified in the papers. Quantization is also generally unmentioned, although this may simply indicate that it was not used and thus is less problematic. The authors of the HumanEvalFix benchmark [1] used a decoding setup with temperature $T=0.2$, $top\_p=0.95$, and generated 20 samples to estimate pass@1 performance. Some other authors adopt these settings in their setups [16,20,22,27]. Some papers report using greedy decoding, without further details [21,23,25]. A few papers explicitly mention using the Code Generation LM Evaluation Harness [21,22,25,27], the framework released by the benchmark’s original team. As for the prompts used for evaluation on HumanEvalFix, most authors report their settings [1,16,21,22,23,25,27]. However, some papers have major shortcomings in reporting any of the evaluation settings described above for this benchmark [17,24,30].

For our local evaluations, we selected models that appear in at least two papers and also exhibited notable differences in benchmark results. Models larger than 15B parameters were excluded due to hardware limitations. While our main focus is on instruction-tuned models—as these are more frequently evaluated throughout studies—, we also take the results of the base models into consideration to assess whether they reproduce results originally attributed to their instruction-tuned counterparts. Consequently, although CodeLlama-7B does not meet the inclusion criteria, we do include it in our evaluations because it serves as the base model for CodeLlama-7B-Instruct. The selected models are as follows: DeepSeekCoder (1.3B and 6.7B), DeepSeekCoder-Instruct (1.3B and 6.7B), CodeLlama (7B and 13B), CodeLlama-Instruct (7B and 13B), CodeGemma (2B and 7B), CodeGemma-Instruct (7B), and WizardCoder (15B). We evaluated these models on the HumanEvalFix benchmark, using the following evaluation settings and values:

• Prompt template: The evaluation framework defaults to the instruct prompt template, with only the context and instruction (the program and the instruction to fix it). The default prompt template lacks the model-specific formatting suggested by model authors. We evaluate models both with their suggested prompt template as well as the default instruct setting. All prompt settings used in our study are detailed in Appendix A.1.
• Sampling vs. greedy decoding: The default behavior in the framework is not to use greedy decoding, but to apply sampling with the temperature $T=0.2$. Alongside greedy decoding, we conduct experiments using sampling with temperatures $T=0.2$ and $T=0.8$.
• Limiting generation length: The default value of 512 tokens is insufficient, as it can lead to unfinished programs and reduced benchmark scores. We conducted our experiments using lengths of 512 and 2048. Since the max length generation parameter considers both the prompt and the generated output, it must be large enough to prevent cutting the output short. We found 2048 to be a good choice on this benchmark, as tokenizers typically fit the inputs within 1024 tokens, leaving enough space for the generated output.
• Precision and quantization: The majority of LLMs are released using bf16 precision, making it the preferred choice for precision. We evaluated the models using all three common precision formats: fp16, fp32, and bf16. While quantization is a useful technique to lower memory usage when running models, it is generally best to use models without it. In addition to our tests without quantization, we also ran experiments using 4-bit and 8-bit quantization (with fp16 precision).
• Other settings: Further settings (such as seed), were kept at their default values as defined by the framework for all evaluations. Specifically, the default value for seed is 0.

To assess the effect of each parameter setting individually, we selected a configuration as the baseline. Then, we compared the evaluation results of this baseline configuration with those obtained when only one setting differed. In the baseline setting, the prompt format followed the recommendations of the model’s authors, no sampling was applied, the maximum generation length was set to the sufficiently large value of 2048, and bf16 precision was used without quantization.

Next, we attempted to reproduce the results reported in the literature. To achieve this, we ran an exhaustive grid search for each model, with all possible combinations of the evaluation settings. As we had two options for prompt, three for sampling and temperature, two for generation length, and five for precision and quantization, we evaluated every single model in 60 settings. Although some reports originated from generating $n=20$ samples with $T=0.2$ to provide a more refined estimation of pass@1 performance, it would be too time-consuming for our grid search approach.

The experiments were performed on A100 GPUs with 40GB VRAM each. We set max_memory_per_gpu to the value of auto, which distributes the model on the assigned GPUs. We utilized only as many GPUs as required to load the model and execute the benchmark, which is typically just one. Where multiple GPUs were used, we also measured whether modifying the number of assigned GPUs had an influence on results; we did not see any variation in benchmark results when increasing the number of utilized GPUs.

4. Experimental Results

In this section, we present the results of our evaluations. First, we examined the impact of each evaluation parameter individually, compared to the baseline configuration. Next, we performed an exhaustive grid search for each model and assessed whether the reported results could be reproduced. All evaluation results are summarized in Figure 2.

4.1. The Effect of Individual Evaluation Settings

We measured the effect of individual evaluation settings by comparing the baseline configuration against variants in which evaluation parameters are modified individually. The effect of each parameter change on the benchmark scores was analyzed in isolation. The impact of these modifications is visualized in Figure 3.

4.1.1. Temperature: Greedy Evaluation and Sampling

Considering all models, sampling with temperature $T=0.2$ has a slight impact on the average performance: $-0.82\%$. The change in performance is more significant when using $T=0.8$: $+5.48\%$. However, by only considering the instruction-tuned models, this change is less notable: $+0.34\%$ (with $T=0.2$) and $+1.34\%$ (with $T=0.8$).

For the baseline of greedy decoding, the standard deviation of scores across precision and quantization settings is 1.49 for all models and 1.74 for instruction-tuned models. When we switch to sampling at $T=0.2$, the standard deviation rises to 1.86 ($+25.19\%$) for all models and 2.23 ($+28.70\%$) for the instruction-tuned ones. Increasing the temperature further to $T=0.8$ increases the standard deviation even more, to 2.36 ($+58.54\%$) for all models and 2.89 ($+66.64\%$) for the instruction-tuned models.

4.1.2. Maximum Generation Length

In the case of DeepSeekCoder-Instruct (1.3B) (evaluated using fp16), we see a notable decrease in performance ($21.34\%\to 16.46\%$) when limiting the length of generation to the default value (to 512). This is a significant effect with a relative drop of $22.87\%$. The effect also applies more generally: across all models, the average performance drop is $19.45\%$. Similarly, for the instruction-tuned models, this drop is $19.80\%$.

4.1.3. Using an Incorrect Prompt Template

Considering the instruction-tuned models, switching from the suggested template to the default instruct prompt template resulted in average relative drop of $6.71\%$ in model performance. This phenomenon is not observable when base models are also included in the calculation: the average performance even increases by $1.64\%$.

4.1.4. Precision and Quantization

By switching from bf16 to fp32 or fp16 precision, the performance drops by $2.19\%$ (fp32) and $1.64\%$ (fp16) for all models and increases by $1.01\%$ (fp32) and $2.01\%$ (fp16) for instruction-tuned models. 8-bit quantization seems to have a similarly small effect with changes of $-0.82\%$ (all models) and $+3.02\%$ (instruction-tuned). However, using 4-bit quantization decreases the overall scores by a significant margin: $-11.23\%$ for all models and $-6.71\%$ for the instruction-tuned ones.

4.2. Reproducibility of Results in Existing Research

We attempted to reproduce existing results in the literature by trying out all possible combinations of the experimental settings. Our main goal was to identify potential misconfigurations in the original setups. All results from our evaluations as well as the scores from the existing studies can be seen in Figure 2.

4.2.1. CodeLlama

The results reported in studies for CodeLlama-Instruct (7B) range from $15.2\%$ to $28.0\%$. This is in line with our evaluations, in which this model achieves scores from $14.02\%$ to $25.61\%$. We reproduced all of the reported scores with one exception, indicating that the differences in reported results are indeed caused by variations in the evaluation settings. For CodeLlama-Instruct (13B), the reported results range from $15.2\%$ to $30.5\%$. Our evaluation yielded considerably lower scores: $10.37\%$ to $23.17\%$. We reproduced four out of six results.

The base CodeLlama (13B) model has two reported scores across papers: $6.1\%$, which we reproduced, and an unexpectedly high score of $33.1\%$. The latter score is very close to the upper range of scores reported for CodeLlama-Instruct (13B).

4.2.2. DeepSeekCoder

The reported results for the DeepSeekCoder (1.3B and 6.7B) base models are as high as $16.4\%$ and $45.4\%$ for the 1.3B and 6.7B variants, respectively. These scores are unusually high, especially when considering our own evaluations, which never exceed $11.59\%$ (1.3B) and $35.37\%$ (6.7B).

DeepSeekCoder (1.3B), for example, is measured to have both a $1.2\%$ and $16.4\%$ pass@1 score, which represents more than a 10× difference. We evaluated this model and its instruction-tuned variant and reproduced both reported results: $1.2\%$ was reproduced using the base model, while the score of $16.4\%$ was only reproducible with the instruction-tuned variant. Using the 6.7B base model, we reproduced two of the reported results, but not the very high score of $45.4\%$. Its instruction-tuned variant however, has yielded multiple results close to this value, with the closest being $45.12\%$.

For the instruction-tuned 1.3B model, one of three values was reproduced. The score of $9.1\%$ might come from the base model, as some of our evaluations yielded values close to it. Our evaluation results of the 6.7B instruction-tuned model range from $31.71\%$ and $52.44\%$. These are partially in line with the reported results, with two very close results.

4.2.3. CodeGemma

Our evaluations of the CodeGemma (2B) base model attained up to $12.20\%$ for the 2B model with one of the two reported scores reproduced. Using the 7B base model, values reach $17.07\%$ with one exact match and one near match to the reported scores; the third score was significantly higher. In our evaluations of CodeGemma-Instruct (7B), we reproduced one of the two reported results.

4.2.4. WizardCoder

Our local evaluations of WizardCoder (15B) were close to one of the reported values ($31.8\%$), with five near matches. However, our evaluations did not achieve even close to the other reported value of $51.6\%$.

5. Discussion

The results of our reproducibility experiments highlight several possible reasons for the discrepancies between the results in the literature. They also point to some strategies that could help prevent these discrepancies.

In some cases, we see surprisingly large scores for base models. We think this is due to confusing the base model with its instruction-tuned variant. It is crucial to ensure that the intended model variant is used, not one with different tuning. Furthermore, it is necessary to confirm that the correct model name and type is specified when reporting the evaluation results.

The pass@1 scores do not change significantly when switching from greedy evaluation to sampling, especially for instruction-tuned models. However, the standard deviation increases, especially with higher temperature values. Thus, we observe that greedy evaluation should be preferred over sampling when calculating the pass@1 performance on this task.

Restricting the models to a short generation length affects the results negatively, resulting in large drops in performance. In order to obtain accurate results, a proper limit should be chosen to avoid cutting possibly correct generations. In the case of our benchmark of focus, HumanEvalFix, 2048 is such a value, but this might differ for other benchmarks.

Using the proper prompt template (defined by model authors) is necessary to ensure stable and reliable evaluation results. Without it, the evaluation may not reflect accurate results. This only applies to instruction-tuned models, as base models were not fine-tuned to process such templates.

We noticed only a slight variation when modifying precision settings or using 8-bit quantization, suggesting these factors do not strongly account for differing results across papers. However, 4-bit quantization does have a notable effect.

The observed inconsistencies in the HumanEvalFix benchmark scores raise the question of whether other benchmarks are affected by similar issues. While related work has suggested that this is indeed the case [9,10,11,12], we further support this claim through an additional small-scale study. To this end, we reviewed further literature to investigate whether benchmark result inconsistencies also appear more broadly. We specifically focused on MBPP and its improved variant, MBPP+. We limited our investigation to DeepSeekCoder models. The reported scores across papers are summarized in

Table 2. Reported MBPP and MBPP+ results of DeepSeekCoder variants.

Model	MBPP	MBPP+
DeepSeekCoder-Inst (33B) [3]	61 [36], 66 [37], 73.2 [22], 78.7 [38,39], 80.4 [40,41,42]	59.1 [22], 66.7 [38,39], 70.1 [40,41,42]
DeepSeekCoder-Inst (6.7B) [3]	60.8 [36], 70.2 [22], 72.7 [38], 73.2 [39], 74.3 [43], 74.9 [40,41,42]	56.6 [22], 63.4 [38,39], 65.1 [43], 65.6 [40,41,42]
DeepSeekCoder-Inst (1.3B) [3]	55.4 [22], 63.7 [38], 65.3 [41]	46.9 [22], 53.1 [38], 54.8 [41]
DeepSeekCoder (33B) [3]	74.2 [41,42]	60.7 [41,42]
DeepSeekCoder (6.7B) [3]	70.2 [38,39,41,42]	51.6 [42], 56.6 [38,39,41], 66.4 [44]
DeepSeekCoder (1.3B) [3]	55.4 [38], 55.6 [41]	46.9 [38,41]

. The discrepancies in these scores indicate that challenges in benchmark reproducibility extend beyond bugfixing on the HumanEvalFix benchmark.

To ensure reproducibility and comparability in future research, we propose the following standardized evaluation checklist:

1.

Use the same precision as specified in the original model release (bf16 for most models) or possibly fp32 to reduce the probability of rounding errors.

2.

Avoid model quantization if possible.

3.

Follow the prompt formatting recommended by model authors.

4.

Unless stated otherwise by the model authors, use greedy decoding for generation stability or generate more samples (such as $n=20$) to provide a more refined pass@1 estimate.

5.

Ensure that limits to generation length are sufficiently large to allow complete responses.

6.

Report the exact model name, version, tuning, number of parameters.

7.

Provide details about the used evaluation framework, evaluation settings, and benchmark dataset version.

6. Conclusions

In this paper, we highlighted the issue of inconsistent benchmark results in the literature on the HumanEvalFix benchmark, one of the most widely used benchmarks for evaluating bugfixing. We found that for most of the models mentioned in multiple papers, the reported benchmark results can vary significantly. We conducted evaluations to determine the effect of various evaluation settings and to reproduce different scores reported for the same models, in order to uncover the reasons for these inconsistencies.

Through a series of empirical evaluations, using multiple models in different sizes and tunings, we identified the set of factors influencing benchmark results. Our experiments revealed that factors such as the prompt template, maximum generation length, and decoding strategies had a notable influence on the benchmark results, whereas precision and 8-bit quantization did not. We have also found cases where results were likely misattributed between instruction-tuned and base models. We wish to emphasize the importance of using appropriate evaluation settings and including these settings in the studies to ensure reliable and reproducible results.

• Broader Impact   

We hope that by giving a clearer picture of the capabilities of LLMs, our research will have the positive impact of helping both academic and industrial stakeholders choose more appropriate models. Collectively, these decisions could influence both academic and commercial adoption of LLMs towards a more evidence-based and better direction.

Our environmental impact was relatively small: the experiments consumed roughly 100 GPU-hours on two Nvidia A100 GPUs. We believe that even this impact could be offset by other researchers not having to waste GPU cycles chasing irreproducible results.