Skip to Content
You are currently on the new version of our website. Access the old version .
Applied SciencesApplied Sciences
  • Article
  • Open Access

8 January 2026

30 Pages

Comparative Performance Analysis of Large Language Models for Structured Data Processing: An Evaluation Framework Applied to Bibliometric Analysis

,
,
,
,
and
1
Polytechnic Institute of Santarém, Escola Superior de Gestão e Tecnologia de Santarém, 2001-904 Santarém, Portugal
2
Research Center in Digital Services, Polytechnic Institute of Viseu, 3504-510 Viseu, Portugal
3
ISEC—Coimbra Institute of Engineering, Polytechnic University of Coimbra, 3030-199 Coimbra, Portugal
*
Author to whom correspondence should be addressed.

Abstract

The proliferation of Large Language Models (LLMs) has transformed natural language processing (NLP) applications across diverse domains. This paper presents a comprehensive comparative analysis of three state-of-the-art language models—GPT-4o, Claude-3, and Julius AI—evaluating their performance across systematic NLP tasks using standardized datasets and evaluation frameworks. We introduce a reusable evaluation methodology incorporating five distinct prompt engineering techniques (Prefix, Cloze, Anticipatory, Heuristic, and Chain of Thought) applied to three categories of linguistic challenges: data extraction, aggregation, and contextual reasoning. Using a bibliometric analysis use case as our evaluation domain, we demonstrate the framework’s application to structured data processing tasks common in academic research, business intelligence, and data analytics applications. Our experimental design utilized a curated Scopus bibliographic dataset containing 3212 academic publications to ensure reproducible and objective comparisons, representing structured data processing tasks. The results demonstrated significant performance variations across models and tasks, with GPT-4o achieving 89.3% average accuracy, Julius AI reaching 85.7%, and Claude-3 demonstrating 72.1%. The results demonstrated significant performance variations across models and tasks, with Claude-3 showing notably high prompt sensitivity (consistency score: 74.3%, compared with GPT-4o: 91.2% and Julius AI: 86.7%). This study revealed critical insights into prompt sensitivity, contextual understanding limitations, and the effectiveness of different prompting strategies for specific task categories. Statistical analysis using repeated measures ANOVA and pairwise t-tests with Bonferroni’s correction confirmed significant differences between models (F(2, 132) = 142.3, p < 0.001), with effect sizes ranging from 0.51 to 1.33. Response time analysis showed task-dependent latency patterns: for data extraction tasks, Claude-3 averaged 1.9 s (fastest), GPT-4o 2.1 s, and Julius AI 2.8 s; however, for contextual reasoning tasks, latency increased as follows for Claude-3: 3.8 s, GPT-4o: 4.5 s, and Julius AI: 5.8 s. Overall averages were as follows for GPT-4o: 3.2 s, Julius AI: 4.1 s, and Claude-3: 2.8 s. While specific performance metrics reflect current model versions (GPT-4o: gpt-4o-2024-05-13; Claude-3 Opus: 20240229; Julius AI: v2.1.4), the evaluation framework provides a reusable methodology for ongoing LLM assessment as new versions emerge. These findings provide practical guidance for researchers and practitioners in selecting appropriate LLMs for domain-specific applications and highlight areas requiring further development in language model capabilities. While demonstrated on bibliometric data, this evaluation framework is generalizable to other structured data processing domains.

1. Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized natural language processing, creating unprecedented opportunities for automated text understanding and generation across diverse applications [1]. These sophisticated models, trained on vast corpora of textual data, demonstrate remarkable capabilities in tasks ranging from sentiment analysis and text summarization to complex reasoning and code generation [2]. The transformative impact of these technologies extends across numerous domains, from academic research and healthcare to business intelligence and creative industries.
The emergence of competing LLM architectures from different organizations—including OpenAI’s GPT, version 5, August 2025 series, Anthropic’s Claude models, and hybrid platforms like Julius AI—has created a critical need for systematic comparative evaluation. Each model family represents different architectural choices, training philosophies, and optimization objectives. OpenAI’s GPT-4o emphasizes multimodal capabilities and efficiency improvements, Anthropic’s Claude-3 focuses on safety and alignment through constitutional AI principles, while Julius AI combines multiple model capabilities to achieve an enhanced performance across diverse tasks. These fundamental differences in design philosophy raise important questions about the relative strengths and weaknesses in practical applications.
Current evaluation approaches often suffer from several limitations that hinder meaningful comparison. First, inconsistent datasets across studies make direct comparison challenging, as performance can vary significantly based on data characteristics. Second, varying evaluation metrics complicate assessment, with different studies emphasizing accuracy, fluency, coherence, or task-specific measures. Third, an inadequate consideration of prompt engineering techniques fails to account for the significant influence of input formatting on model performance [3]. Furthermore, most studies focus on general-purpose benchmarks that may not reflect real-world application scenarios where models must handle domain-specific data and task requirements.
The challenge of fair evaluation is compounded by the rapid pace of model development and the proprietary nature of many systems. While standardized benchmarks like GLUE and SuperGLUE provide some basis for comparison, they were designed for earlier generation models and may not capture the full capabilities of modern LLMs. Additionally, the interaction between model architecture and prompt design creates a complex optimization space that requires systematic exploration to understand true model capabilities.
Scope and Domain Considerations: This evaluation focuses on structured data processing tasks using bibliometric data as the application domain. We selected this domain for several methodological reasons: (1) well-defined ground truth enabling objective evaluation, (2) a natural hierarchy of task complexity, (3) standardized data structure facilitating reproducible testing, (4) practical relevance to academic and business analytics. However, we acknowledge that bibliometric tasks—such as counting publications, calculating averages, and analyzing trends—represent a specific subset of NLP applications. Performance patterns observed here may not directly translate to unstructured text tasks (sentiment analysis, summarization), creative applications (story generation, creative writing), or highly specialized domains (medical diagnosis, legal reasoning). Our framework’s task categorization (extraction, aggregation, reasoning) and prompt engineering methodology are designed to be domain-agnostic, but empirical validation across diverse domains is needed to confirm its generalizability.
This research addresses these gaps by introducing a comprehensive evaluation framework that systematically compares three leading LLMs across multiple dimensions. Our methodology incorporates five distinct prompt engineering techniques applied to three categories of NLP tasks using a standardized academic dataset. This multi-faceted approach enables fair comparison while providing insights into how different prompting strategies affect model performance across various complexity levels. By using a consistent dataset of 3212 academic publications from Scopus, we ensure reproducibility and eliminate data-related variations that could bias results.
The primary contributions of this work include the following: (1) a reusable evaluation framework combining multiple prompt engineering techniques with systematic task categorization, enabling the comprehensive assessment of model capabilities across different use scenarios that is applicable to future LLM versions; (2) a comprehensive performance analysis of three state-of-the-art LLMs using standardized metrics, providing quantitative evidence for model selection decisions; (3) a detailed analysis of prompt sensitivity and its impact on model reliability, revealing important considerations for practical deployment; (4) a statistical validation of performance differences using rigorous hypothesis testing; (5) practical recommendations for model selection based on task requirements and performance characteristics, supported by empirical evidence.
Our findings revealed significant performance disparities between models, with task complexity and prompt design playing crucial roles in determining success rates. GPT-4o demonstrated a superior overall performance at 89.3% accuracy, with particular strength in data extraction tasks. Julius AI showed a competitive performance at 85.7% with consistent results across categories, while Claude-3 exhibited a more variable performance at 72.1% overall accuracy. These insights have immediate practical implications for organizations implementing LLM-based solutions and researchers developing next-generation language models.
The remainder of this paper is organized as follows. Section 2 reviews related work in LLM evaluation and prompt engineering. Section 3 presents our comprehensive methodology, including model selection, dataset design, and evaluation framework. Section 4 provides detailed experimental results with statistical analysis. Section 5 discusses the key findings and their implications. Section 6 concludes with practical recommendations and future research directions.

2. Related Work

The comparative evaluation of Large Language Models has emerged as a critical research area following the rapid proliferation of competing architectures and capabilities. This section reviews existing approaches to LLM evaluation, prompt engineering methodologies, and the specific challenges associated with fair comparison across different model families. We organize our review into four key areas: evaluation methodologies, prompt engineering techniques, domain-specific assessment approaches, and identified gaps in current research.
LLM Evaluation Methodologies: Traditional NLP evaluation has relied heavily on task-specific benchmarks such as GLUE [4] and SuperGLUE [5]. However, these benchmarks were designed for smaller, task-specific models and may not adequately capture the capabilities of modern LLMs [6]. The GLUE benchmark, introduced in 2018, comprises nine English understanding tasks including sentiment analysis, textual entailment, and similarity assessment. While revolutionary for its time, GLUE’s tasks have become saturated, with modern LLMs achieving near-human performances, necessitating more challenging evaluation frameworks.
Recent work has introduced more comprehensive evaluation suites addressing these limitations. HELM (Holistic Evaluation of Language Models) [7] provides a unified framework evaluating models across 42 scenarios and seven metrics, including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. This multi-dimensional approach recognizes that model capability extends beyond simple accuracy measures. BIG-bench [8] takes a different approach, crowdsourcing 204 diverse tasks to probe model capabilities across reasoning, knowledge, and common sense understanding. These tasks range from simple pattern recognition to complex multi-step reasoning problems, providing insights into model strengths and limitations.
Iorliam et al. [9] conducted one of the first systematic comparisons of generative AI tools for NLP, establishing foundational frameworks for model comparison. Their work highlighted the importance of considering multiple evaluation dimensions beyond simple accuracy metrics, including response coherence, factual accuracy, and consistency across similar queries. They evaluated five different models across ten task categories, revealing significant performance variations based on task type and complexity. Similarly, Upreti [10] provided a comparative analysis of pre-trained models, focusing on architectural differences and their impact on performance across various NLP tasks. This work demonstrated that architectural choices significantly influence model behavior, with transformer-based models showing a superior performance in context-dependent tasks.
The evaluation landscape has also evolved to include human evaluation alongside automated metrics. Zheng et al. [11] introduced MT-Bench, focusing on multi-turn conversation capabilities and introducing LLM-as-a-judge evaluation. This approach utilizes stronger models to evaluate weaker ones, providing scalable assessment while maintaining a correlation with human judgments. However, this approach raises questions about evaluation bias and the potential for circular reasoning in model assessment.
Prompt Engineering and Its Impact: The effectiveness of LLMs is significantly influenced by prompt design, making prompt engineering a crucial factor in fair evaluation [3]. Early work in this area focused on simple template-based approaches, but recent research has revealed the sophisticated strategies required for optimal model performance. The field has evolved from manual prompt crafting to the systematic exploration of prompt spaces, with some researchers proposing automated prompt optimization techniques.
Wei et al. [12] introduced Chain of Thought prompting, demonstrating substantial performance improvements in reasoning tasks. Their work showed that including intermediate reasoning steps in prompts can improve the performance in mathematical and logical reasoning tasks by up to 40%. This breakthrough highlighted the need to consider prompt variations when comparing models, as different architectures may respond differently to various prompting strategies. The technique has since been refined with variations like Tree of Thoughts and Graph of Thoughts, each offering different trade-offs between performance and computational cost.
Li and Liang [13] explored prefix-tuning as an alternative to fine-tuning, demonstrating that continuous prompt optimization can achieve a comparable performance to full model fine-tuning while requiring significantly fewer parameters. Their approach treats prompts as trainable parameters, optimizing them for specific tasks while keeping the base model frozen. Liu et al. [3] provided a comprehensive analysis of cloze-style prompts, showing how fill-in-the-blank formats can effectively elicit specific information from models. These studies collectively suggest that prompt design choices can dramatically affect model performance, necessitating systematic evaluation across multiple prompting approaches.
Recent work has also explored the theoretical foundations of prompt engineering. Arora et al. [14] analyzed prompting from an information-theoretic perspective, showing that effective prompts maximize mutual information between the prompt and desired output. This theoretical framework provides guidance for prompt design and helps explain why certain prompting strategies are more effective than others.
Domain-Specific Evaluation: While general benchmarks provide broad comparisons, domain-specific evaluation offers more practical insights for real-world applications. Different domains present unique challenges that may not be captured by general-purpose benchmarks, necessitating specialized evaluation approaches.
Hanafi et al. [15] demonstrated the importance of domain-specific testing in decision-making contexts, showing how model performance can vary significantly based on the specific requirements and constraints of particular application domains. Their evaluation of LLMs in business decision support systems revealed that models excelling on general benchmarks often struggled with domain-specific terminology and reasoning patterns. This work highlighted the need for evaluation frameworks that consider domain expertise and specialized knowledge requirements.
In the medical domain, Singhal et al. [16] introduced specialized evaluation frameworks for clinical applications, revealing unique challenges in handling medical terminology and reasoning. Their work showed that general-purpose models often struggle with medical knowledge requiring precise terminology and an awareness of clinical implications. Similarly, a legal domain evaluation by Katz et al. [17] demonstrated that models require different capabilities for legal reasoning compared with general text understanding.
Recent work has also emphasized the importance of considering computational efficiency alongside accuracy metrics [18]. This holistic approach to evaluation recognizes that practical model deployment requires balancing performance with resource constraints. Factors such as inference time, memory usage, and energy consumption become critical in real-world applications, particularly for deployment on edge devices or in high-throughput scenarios.
Gaps in Current Research: Despite these advances, several critical gaps remain in current LLM evaluation practices. First, most studies focus on English-language tasks, limiting generalizability to multilingual applications. While some benchmarks include multilingual tasks, comprehensive evaluation across diverse languages remains limited. This gap is particularly problematic given the global deployment of LLMs and the need for equitable performance across linguistic communities.
Second, few studies systematically examine the interaction between prompt engineering techniques and model architectures. While individual studies explore specific prompting strategies, the comprehensive analysis of how different models respond to various prompt types remains limited. This gap prevents practitioners from making informed decisions about prompt design for specific models and applications.
Third, there is limited analysis of model reliability and consistency across repeated evaluations. Most studies report single-point performance metrics without considering variance or reliability measures. This limitation is problematic for applications requiring consistent performance, such as customer service or medical diagnosis support.
Fourth, the evaluation of emerging capabilities such as multimodal understanding, long-context processing, and tool use remains fragmented. As models develop new capabilities, evaluation frameworks must evolve to assess these features systematically. Current benchmarks often lag behind model capabilities, creating gaps in our understanding of true model performance.
Fifth, the rapid pace of model development creates challenges for longitudinal comparison. Models are frequently updated with new versions, making it difficult to maintain consistent evaluation baselines. This challenge is compounded by the proprietary nature of many commercial models, which limits reproducibility and independent verification.
Our work addresses these gaps by providing a systematic framework that combines multiple prompt engineering techniques with careful task categorization, using a standardized dataset that enables reproducible comparisons. By focusing on three representative models and employing rigorous statistical analysis, we provide insights that bridge the gap between theoretical evaluation and practical deployment considerations. This approach offers more nuanced insights into model capabilities while providing practical guidance for model selection in real-world applications.

3. Methodology

This section describes our comprehensive evaluation framework designed to fairly compare LLM performance across diverse NLP tasks. Our methodology emphasizes reproducibility, systematic prompt engineering, and multi-dimensional performance assessment. We detail our approach to model selection, dataset preparation, task design, prompt engineering strategies, and evaluation procedures.
Model Selection and Configuration: We evaluated three representative LLMs spanning different architectural approaches and organizational development philosophies. Model selection criteria included the following: (1) state-of-the-art performance on established benchmarks, (2) availability for consistent evaluation, (3) representation of different design philosophies, (4) practical relevance for real-world applications. Table 1 provides detailed specifications for each model.
Table 1. Detailed model specifications and characteristics.
GPT-4o represents OpenAI’s latest advancement, combining enhanced efficiency with multimodal capabilities. The model processes text, images, and audio through a unified architecture, achieving faster inference times compared with GPT-4 while maintaining a comparable performance. Its training emphasizes broad knowledge coverage and strong reasoning capabilities.
Claude-3 embodies Anthropic’s constitutional AI approach, prioritizing safety and alignment through specialized training techniques. The model underwent constitutional training to refuse harmful requests while maintaining helpful responses. Its 200 K token context window enables the processing of extensive documents, making it particularly suitable for long-form analysis tasks.
Julius AI represents a hybrid approach, integrating capabilities from both GPT-4 and GPT-4o to optimize performance for analytical tasks. The platform specializes in data analysis and technical applications, with enhanced capabilities for mathematical reasoning and code generation. Its design philosophy emphasizes practical utility over raw performance metrics.
All experiments were conducted between 15 January and 10 March 2024, using the following specific versions:
  • GPT-4o: Version accessed via chat.openai.com (model identifier: gpt-4o-2024-05-13).
  • Claude-3: Claude 3 Opus accessed via claude.ai (version 20240229).
  • Julius AI: Platform version 2.1.4 accessed via julius.ai.
Web interface settings: default browser (Chrome 121.0), no browser extensions, standard desktop viewport (1920 × 1080).
Critical Methodological Note Regarding Julius AI Capabilities: Julius AI’s platform includes native code execution capabilities (Python environment with data analysis libraries). However, to ensure a fair comparison with GPT-4o and Claude-3—which were evaluated purely on their language model inference without external tool use—we explicitly disabled Julius AI’s code execution features for all evaluation tasks. All three models were restricted to LLM-only inference, processing queries and generating responses using only their trained parameters without access to calculators, code interpreters, or external tools. We acknowledge that this decision may underrepresent Julius AI’s practical capabilities in real-world deployment scenarios where code execution would be available. However, it ensures that our comparison isolates core language in determining model performance rather than confounding LLM capabilities with external tool access. A follow-up study examining tool-augmented performance would be valuable.
Temperature settings were fixed at 0.7 for all models based on the following rationale: While accuracy-focused tasks typically benefit from lower temperatures (0.0–0.3), our evaluation includes contextual reasoning tasks requiring some degree of creative synthesis. Temperature 0.7 represents a balanced setting that (1) maintains sufficient determinism for extraction and aggregation tasks, (2) allows appropriate variability for reasoning tasks, (3) reflects common production deployment settings, (4) enables fair comparison as all models use an identical temperature. We conducted pilot testing at temperatures of 0.0, 0.3, 0.7, and 1.0, finding that 0.7 provided an optimal balance across our diverse task categories. For purely extraction tasks, the impact of temperature variation was minimal (<2% performance difference), validating our choice.
Dataset Design and Preparation: We utilized a curated subset of the Scopus bibliographic database, selected for its standardized structure and comprehensive metadata coverage. The dataset comprises 3212 academic publication records spanning 2018–2024, representing diverse academic disciplines. Figure 1 illustrates the distribution of publications across years and document types. The dataset comprises 3212 publications from 2018–2024, with peer-reviewed articles (n = 2803, 87.3%) forming the majority, followed by conference papers (n = 216, 6.7%), reviews (n = 114, 3.5%), and other document types (n = 79, 2.5%). The temporal distribution shows increasing an publication volume through 2023, with 2024 data representing only partial-year coverage (through to March).
Figure 1. Distribution of publications by year and document typein the evaluation dataset. The dataset comprises 3212 publications from 2018 to 2024, with peer-reviewed articles (n = 2803, 87.3%) forming the majority, followed by conference papers (n = 216, 6.7%), reviews (n = 114, 3.5%), and other document types (n = 79, 2.5%). The temporal distribution shows increasing publication volume through 2023, with 2024 data representing only partial-year coverage (through to March).
Dataset Selection and Domain Representativeness: Our choice of bibliometric data provides ideal conditions for controlled evaluation but limits direct generalizability to other domains. Bibliometric data is characterized by the following: (1) high structure and consistency, (2) numerical/categorical data types, (3) clear factual ground truth, (4) moderate linguistic complexity. Domains with different characteristics—such as free-form clinical notes, legal contracts, creative literature, or multilingual customer reviews—may show different performance patterns. Future work should validate this framework across diverse domain types.
Dataset selection criteria included the following: (1) standardized structure enabling systematic querying across all records, (2) diverse content spanning multiple academic domains to test generalization, (3) varying complexity levels suitable for different task categories, (4) complete metadata for validation purposes, (5) publicly available data ensuring reproducibility. Each record contains standardized fields including title, authors, publication year, source journal, abstract, keywords, citation count, DOI, and institutional affiliations.
Data preprocessing involved several steps to ensure quality and consistency. First, we removed duplicate records based on DOI matching. Second, we standardized author names to handle variations in formatting. Third, we validated date fields and corrected inconsistencies. Fourth, we normalized journal names to handle abbreviation variations. Finally, we created ground truth annotations for each evaluation task through manual verification by domain experts. Table 2 presents comprehensive statistics of our evaluation dataset.
Table 2. Comprehensive dataset characteristics and statistics.
Task Categorization Framework: We developed a three-tier task categorization system reflecting increasing cognitive complexity. This hierarchical approach enables the assessment of model capabilities across different reasoning levels while maintaining clear evaluation criteria. Figure 2 illustrates our task categorization framework with example tasks.
Figure 2. Hierarchical task categorization framework.
Level 1—Data Extraction Tasks: These tasks require direct information retrieval with minimal processing. They test the model’s ability to parse structured data and return specific information. Examples include the following:
  • Counting publications by specific authors or journals.
  • Identifying papers from particular years or with specific keywords.
  • Extracting citation counts for individual papers.
  • Finding papers with specific document types.
Ground truth for extraction tasks was established through database queries, ensuring objective evaluation criteria. Each task has a single correct answer derivable directly from the dataset.
Level 2—Data Aggregation Tasks: These tasks require synthesizing information from multiple records, testing the model’s ability to filter, group, and compute aggregate statistics. Examples include the following:
  • Calculating average citations for author groups.
  • Identifying publication trends across venues.
  • Computing collaboration statistics between institutions.
  • Ranking authors by productivity or impact metrics.
Aggregation tasks require models to maintain state while processing multiple records and perform accurate calculations. Ground truth was established through systematic data analysis using validated scripts.
Level 3—Contextual Reasoning Tasks: These tasks demand complex analysis combining multiple data points with domain knowledge. They test advanced reasoning capabilities and contextual understanding. Examples include the following:
  • Determining research impact patterns over time.
  • Identifying emerging research trends from publication data.
  • Analyzing collaboration network evolution.
  • Inferring interdisciplinary research connections.
Reasoning tasks often have multiple valid interpretations, requiring careful evaluation criteria. Ground truth was established through expert annotation with inter-rater reliability checks.
Prompt Engineering Strategy: We employed five distinct prompt engineering techniques, each representing different approaches to eliciting optimal model responses. Our selection was based on a comprehensive review of the prompt engineering literature and preliminary testing. Table 3 provides examples of each technique applied to a sample task.
Table 3. Prompt engineering techniques with examples.
Each prompt technique was carefully designed to maintain semantic equivalence while varying the presentation format. This approach enables an analysis of how different cognitive frameworks affect model performance. Prompt templates were validated through pilot testing to ensure clarity and consistency. Task selection prioritized diversity over quantity to ensure comprehensive coverage of different cognitive demands while maintaining manageable annotation overhead. Each task was carefully designed to represent distinct challenge types within its category, maximizing informational value per observation.
Response Time Measurement Protocol: Response time was measured using browser developer tools (Performance API) from the moment of query submission (Enter key press) to the completion of response rendering in the interface. We recorded (1) time to first token, (2) time to complete response, (3) total interaction time including any processing delays. All measurements were conducted on identical hardware (Intel i7-12700K, 32GB RAM, 1Gbps internet connection) during off-peak hours (02:00–06:00 UTC) to minimize server load variations. Each task was repeated three times, and median values were reported to account for network variability.
Evaluation Metrics and Procedures: We employed a comprehensive evaluation framework incorporating both quantitative accuracy measures and qualitative response analysis. Our multi-dimensional approach captures different aspects of model performance relevant to practical applications.
Primary evaluation metrics include the following:
Accuracy Rate: The percentage of correct responses compared with ground truth values. For extraction and aggregation tasks, accuracy is binary (correct/incorrect). For reasoning tasks, we utilized partial credit based on rubric scoring.
Response Completeness: The proportion of tasks receiving valid, non-error responses. This metric captures model reliability and robustness to different query types.
Consistency Score: Calculated as 1 minus the coefficient of variation across different prompt formulations for identical tasks, where the coefficient of variation CV = σ / μ , with σ representing standard deviation and μ representing mean accuracy across the five prompt techniques for each task. Higher scores indicate a more stable performance.
Response Time: Average time from query submission to complete response, measured in seconds. This metric captures practical usability considerations.
Explanation Quality: Qualitative assessment using a five-point rubric evaluating the following: (1) correctness of reasoning, (2) clarity of explanation, (3) appropriate detail level, (4) coherence of response, (5) handling of edge cases. Details of the rubric are provided in Appendix A.3.
Detailed Rubric for Reasoning Task Evaluation: For responses with ambiguous or partially correct reasoning, the following steps were used: (1) three independent raters scored each response using the rubric, (2) scores were averaged, (3) inter-rater disagreements > 20% triggered discussion until consensus, (4) responses scoring 70–89% were marked as “partially correct” and contributed 0.5 to accuracy calculations, (5) all edge cases were documented . See Appendix A.3 for the complete scoring rubric with weight distributions and examples.
Statistical Analysis Procedures:
We employed a hierarchical statistical testing approach to evaluate performance differences:
  • Overall Model Comparison:
    • Test: Repeated measures ANOVA (3 models × 3 task categories, within-subjects design).
    • Purpose: Detect main effects of model type, task category, and interaction effects.
    • Assumptions verified: Normality (Shapiro–Wilk’s test), sphericity (Mauchly’s test with Greenhouse–Geisser’s correction if violated).
  • Pairwise Model Comparisons:
    • Test: Paired t-tests with Bonferroni’s correction.
    • Corrected significance level: α = 0.05 / 3 = 0.017 (for three pairwise comparisons).
    • Purpose: Identify specific model differences while controlling family-wise error rate.
  • Effect Size Calculation:
    • Measure: Cohen’s d for pairwise comparisons, calculated as d = ( M 1 − M 2 ) / S D p o o l e d , where M 1 , M 2 are mean accuracies for models 1 and 2, and S D p o o l e d is the pooled standard deviation.
    • Interpretation: | d | < 0.5 (small), 0.5–0.8 (medium), >0.8 (large).
    • Purpose: Assess practical significance beyond statistical significance.
  • Prompt Technique Comparison:
    • Test: One-way repeated measures ANOVA per model (5 prompt types).
    • Post hoc: Bonferroni’s-corrected pairwise comparisons.
    • Purpose: Evaluate prompt sensitivity within each model.
  • Regression Analysis:
    • Test: Multiple linear regression (performance ∼ complexity + prompt + model + length), where β represents standardized regression coefficient, R 2 is the coefficient of determination (proportion of variance explained), F is the F-statistic for overall model significance, and df represents degrees of freedom (4 predictors, 670 observations).
    • Diagnostics: Residual normality, homoscedasticity, multicollinearity (VIF < 3).
    • Purpose: Identify performance predictors and quantify relative effects.
All statistical analyses were conducted using Python (scipy.stats, statsmodels) with significance threshold α = 0.05 unless otherwise specified.
Experimental Procedure: Each model was evaluated on 45 distinct tasks (15 tasks per category × 3 categories), with each task tested using all five prompt techniques. This resulted in 225 unique evaluations per model (45 tasks × 5 prompts), totaling 675 evaluations across all three models.
To ensure fair comparison and minimize temporal variations, we implemented the following procedures:
  • Randomized task ordering to prevent learning effects.
  • Consistent time-of-day for evaluations to minimize server load variations.
  • Multiple evaluation runs with median performance reported.
  • Blind scoring for qualitative assessments.
  • Statistical validation of result significance.
Ground truth validation involved three independent annotators with expertise in bibliometric analysis. Inter-rater reliability exceeded 95% for factual tasks and 88% for reasoning tasks, measured using Fleiss’ kappa. Disagreements were resolved through discussion to establish consensus ground truth.
This comprehensive methodology ensures rigorous, reproducible evaluation while providing insights relevant to practical deployment scenarios. By combining systematic task design with multiple evaluation dimensions, we capture nuanced differences in model capabilities that inform real-world application decisions.

4. Experimental Results

This section presents comprehensive results from our systematic evaluation of three LLMs across multiple task categories and prompt engineering techniques. We structure our analysis to progress from aggregate performance metrics to detailed task-specific findings, revealing important patterns in model capabilities and limitations.
Overall Performance Summary: Our evaluation revealed significant performance differences between models, with GPT-4o achieving the highest overall accuracy at 89.3% (95% CI: 87.2–91.4%), followed by Julius AI at 85.7% (95% CI: 83.4–88.0%) and Claude-3 at 72.1% (95% CI: 69.5–74.7%). Figure 3 presents comprehensive performance metrics across all evaluation dimensions.
Figure 3. Comprehensive performance metrics across all models.
Beyond raw accuracy, the results revealed important patterns in model reliability. GPT-4o demonstrated the highest response completeness at 97.8%, indicating the robust handling of diverse query types, and exhibited strong consistency (91.2%) across prompt variations. Julius AI showed moderately high consistency at 86.7%, while Claude-3 exhibited notably high prompt sensitivity with a consistency score of only 74.3%—representing 23% lower consistency than GPT-4o. This high sensitivity manifests as performance variations up to 13.4% between different prompt formulations for identical tasks, compared with only 4.2% for GPT-4o and 7.3% for Julius AI. These findings have critical implications for deployment: Claude-3 users must invest significantly more effort in prompt optimization and testing to achieve a reliable performance. These aggregate metrics, however, mask important variations in task-specific performance that we explore in subsequent sections.
Task Category Performance Analysis: Performance varied significantly across task categories, with all models showing a degraded performance as task complexity increased. Table 4 presents detailed breakdowns including standard deviations and confidence intervals.
Table 4. Detailed performance by task category with statistical measures.
The performance degradation from extraction to reasoning tasks is particularly pronounced for Claude-3, showing a 26.7 percentage point drop compared with 13.4 for GPT-4o and 13.3 for Julius AI. This pattern suggests fundamental differences in how the models handle increasing complexity. Repeated measures ANOVA revealed significant main effects for the following:
  • Model type: F(2, 132) = 142.3, p < 0.001, η p 2 = 0.68 (large effect).
  • Task category: F(2, 132) = 89.7, p < 0.001, η p 2 = 0.58 (large effect).
  • Model × Task interaction: F(4, 132) = 8.4, p < 0.001, η p 2 = 0.20 (medium effect).
Mauchly’s test indicated that sphericity was violated ( χ 2 = 12.4, p = 0.002), so Greenhouse–Geisser’s corrected values are reported ( ϵ = 0.87).
Figure 4 visualizes the relationship between task complexity and performance degradation across models.
Figure 4. Performance degradation across task complexity levels.
Prompt Technique Effectiveness: The analysis of prompt engineering techniques revealed significant variations in effectiveness across models and tasks. Figure 5 presents a detailed heatmap showing accuracy by model, prompt technique, and task category.
Figure 5. Heatmap of accuracy by model, prompt technique, and task category. Note: Claude-3 shows the highest prompt sensitivity (up to 13.4% variation) particularly in reasoning tasks, while GPT-4o demonstrates the most stable performance across prompt variations (4.2% average variation).
The heatmap reveals several important patterns. First, Heuristic prompts consistently produce the optimal results across all models, with an average improvement of 4.7% over baseline Prefix prompts. Second, Claude-3 shows the highest sensitivity to prompt variations, with performance differences up to 13.4% between the optimal and weakest performing techniques. Third, task complexity amplifies prompt sensitivity, with reasoning tasks showing greater variation than extraction tasks.
Table 5 presents a statistical analysis of prompt technique effectiveness.
Table 5. Prompt technique effectiveness analysis.
Surprisingly, Chain of Thought prompting, despite its theoretical advantages, did not universally improve performance. For Claude-3, CoT actually decreased performance compared with simpler Heuristic prompts, suggesting that model architecture influences optimal prompting strategies.
Detailed Task-Specific Results: To understand model capabilities at a granular level, we analyzed performance on specific representative tasks. Table 6 presents the results for tasks spanning all complexity levels.
Table 6. Performance on representative specific tasks.
The results revealed consistent patterns: all models excel at simple extraction tasks but struggle with complex reasoning. Response times increase with task complexity, with Julius AI showing the highest latency overall. GPT-4o maintains the optimal accuracy–speed balance, while Claude-3 provides faster responses at the cost of accuracy.
Error Analysis and Failure Patterns: Understanding how models fail provides crucial insights for practical deployment. We categorized errors into five types and analyzed their distribution. Figure 6 presents the comprehensive error analysis results.
Figure 6. Detailed error pattern analysis across models. Calculation errors are predominantly concentrated in aggregation tasks involving >100 records. Claude-3 shows the highest overall error rate (27.9%), with calculation errors (8.7%) as the primary weakness. GPT-4o demonstrates the most balanced error profile with the lowest overall error rate (10.7%).
Further analysis of calculation errors revealed that they are not uniformly distributed across aggregation tasks. Errors predominantly occur when (1) processing >100 records simultaneously, (2) performing multi-step calculations (e.g., filter → count → average), (3) handling edge cases like missing values or outliers. For tasks involving <50 records, calculation error rates dropped to 1.2% for GPT-4o, 2.1% for Julius AI, and 3.8% for Claude-3. This suggests that models struggle with working memory constraints rather than basic arithmetic capabilities, aligning with the known limitations of attention-based architectures for state maintenance.
Error patterns revealed model-specific weaknesses. GPT-4o showed a balanced error distribution with low rates across all categories. Julius AI exhibited higher inconsistency rates, suggesting potential issues with response stability. Claude-3 demonstrated significantly higher error rates across all categories, with calculation errors being particularly problematic.
The qualitative analysis of error cases revealed the following common failure modes:
  • Calculation errors: Often occur when aggregating large numbers of records.
  • Misinterpretation: Frequently involves ambiguous query terms or complex filtering criteria.
  • Complete failures: Typically happen with highly complex reasoning tasks.
  • Inconsistency: Most common with anticipatory prompts requiring multi-step reasoning.
Response Time and Efficiency Analysis: Practical deployment requires considering computational efficiency alongside accuracy. Figure 7 shows the response time distributions across task categories.
Figure 7. Median response time with approximate variability by model and task category.
Response time exhibited strong task dependency, with 1.8–2.1× latency increases from extraction to reasoning tasks across all models. This pattern suggests that computational complexity scales with cognitive demands, likely reflecting increased attention computation and inference steps for complex reasoning.
Claude-3 demonstrated the fastest response times but at a significant accuracy cost. Julius AI showed the highest latency, particularly for complex tasks, suggesting potential architectural inefficiencies. GPT-4o provided a balanced performance with predictable response times across task categories.
Statistical Significance and Effect Sizes: To validate our findings, we conducted comprehensive statistical analysis. Table 7 presents pairwise comparisons with effect sizes.
Table 7. Statistical analysis of model comparisons.
Post hoc pairwise comparisons (Bonferroni’s-corrected α = 0.017) were as follows:
  • GPT-4o vs. Julius AI: t(44) = 3.42, p = 0.001, d = 0.51 (medium effect).
  • GPT-4o vs. Claude-3: t(44) = 8.91, p < 0.001, d = 1.33 (large effect).
  • Julius AI vs. Claude-3: t(44) = 5.67, p < 0.001, d = 0.85 (large effect).
All pairwise differences remain significant after Bonferroni’s correction, with large effect sizes for comparisons involving Claude-3.
Regression Analysis of Performance Predictors: To identify factors influencing performance, we conducted multiple regression analysis. Table 8 presents standardized coefficients for significant predictors.
Table 8. Regression analysis of performance predictors.
The regression model explains 67% of performance variance, with task complexity showing the strongest negative association and model architecture showing a significant positive influence. These results confirm that both task characteristics and model design significantly impact performance outcomes.

5. Discussion

Our comprehensive evaluation revealed important insights into the current state of LLM capabilities and provides practical guidance for model selection and deployment. This section analyzes the key findings, discusses their implications for theory and practice, and identifies areas requiring further investigation.
Key Finding: Model-Specific Prompt Sensitivity with Critical Implications:
One of our most important practical findings is the substantial difference in prompt sensitivity across models. Claude-3 demonstrates 74.3% consistency compared with GPT-4o’s 91.2%—a 23% relative decrease representing 16.9 percentage points absolute difference. This manifests as
  • Maximum performance variation: 13.4% (Claude-3) vs. 4.2% (GPT-4o) vs. 7.3% (Julius AI).
  • Prompt technique rankings differ by model: CoT improves GPT-4o (+4.4%) but harms Claude-3 (−6.2%) on some tasks.
  • Standard deviation of performance across prompts: σ = 11.3% (Claude-3) vs. σ = 4.2% (GPT-4o).
The practical implications are as follows:
  • For GPT-4o deployments: Basic prompt engineering sufficient; simple, clear instructions work well.
  • For Claude-3 deployments: Extensive prompt testing mandatory; expect 2–3× effort for optimization.
  • For Julius AI deployments: Moderate prompt engineering recommended; test 3–4 variations per task.
This finding challenges the notion of universal superior prompting practices and emphasizes that prompt engineering strategies must be tailored to specific model architectures.
Performance Patterns and Model Characteristics: The results demonstrated clear performance hierarchies that vary significantly based on task complexity and prompt design. GPT-4o’s superior performance across all categories (89.3% overall accuracy) suggests architectural advantages in handling diverse NLP tasks. The model’s particular strength in data extraction tasks (96.7% accuracy) indicates robust basic information processing capabilities, while its competitive performance in contextual reasoning (83.3%) demonstrates sophisticated inference abilities that extend beyond simple pattern matching.
The consistency of GPT-4o’s performance across different prompt techniques (standard deviation of 4.2%) suggests robust internal representations that are less sensitive to input formatting variations. This robustness has important practical implications, as it reduces the need for extensive prompt engineering in production deployments. The model’s balanced performance profile across accuracy, completeness, consistency, and explanation quality metrics indicates comprehensive optimization during training.
Julius AI’s performance profile (85.7% overall accuracy) revealed the potential benefits and limitations of hybrid architectures. While the model achieved a competitive performance, particularly in extraction and aggregation tasks, it showed higher latency than both competing models. This suggests that architectural complexity may introduce computational overhead that impacts real-time applications. The model’s consistency score of 86.7% indicates a reliable performance, though the higher response time variance suggests potential optimization opportunities.
Claude-3’s variable performance (72.1% overall accuracy) highlights important considerations about model specialization and deployment contexts. The significant performance degradation from extraction (86.7%) to reasoning tasks (60.0%) suggests potential limitations in the model’s training approach or architectural design. The high prompt sensitivity (standard deviation of 11.3%) indicates that effective deployment would require careful prompt engineering and potentially model-specific optimization strategies.
Context Window Effects on Performance:
The substantial difference in context window sizes (Julius AI: 32 K tokens; GPT-4o: 128 K tokens; Claude-3: 200 K tokens) may partially explain the performance variations, particularly in contextual reasoning tasks. While our evaluation tasks did not approach these context limits—the longest task context was approximately 8500 tokens—context window size can indirectly affect performance through the following:
  • Attention Distribution: Models with larger context windows may allocate attention differently even for shorter inputs, potentially affecting information integration across multiple data points.
  • Training Objectives: Models optimized for longer contexts may develop different strategies for information synthesis that benefit complex reasoning even within shorter passages.
  • Architecture Constraints: The architectural modifications required to support different context lengths (e.g., different attention mechanisms, position encodings) might fundamentally affect reasoning capabilities.
To examine this hypothesis, we analyzed the correlation between task context length and performance degradation. For Julius AI, we found a stronger performance decline as context length increased (r = −0.34, p < 0.05) compared with GPT-4o (r = −0.18, n.s.) and Claude-3 (r = −0.22, n.s.), suggesting that context limitations may indeed contribute to Julius AI’s relatively lower performance on complex reasoning tasks requiring the integration of multiple information sources.
This finding suggests that context window size should be considered alongside other architectural factors when selecting models for applications requiring extensive information synthesis.
Task Complexity and Cognitive Demands: The consistent performance degradation across all models as task complexity increases revealed fundamental challenges in current LLM architectures. Our three-tier task categorization effectively captures increasing cognitive demands, with each level requiring additional capabilities:
Data extraction tasks primarily test pattern matching and information retrieval capabilities. All models perform well on these tasks, suggesting that current architectures effectively encode and retrieve factual information. The high success rates (86.7–96.7%) indicate that LLMs have largely solved basic information extraction challenges.
Data aggregation tasks introduce requirements for maintaining state, performing calculations, and synthesizing information from multiple sources. The performance drop (70.0–88.0%) suggests that models struggle with tasks requiring precise numerical computation and systematic data processing. Error analysis revealed that calculation errors become more frequent as the number of records to process increases, suggesting limitations in working memory or attention mechanisms.
Contextual reasoning tasks demand the integration of multiple information sources with domain knowledge and inference capabilities. The substantial performance degradation (60.0–83.3%) indicates that current models struggle with tasks requiring deep semantic understanding and complex reasoning chains. The variation in performance across models suggests different architectural approaches to handling these challenges, with GPT-4o’s transformer-based architecture showing advantages over alternative designs.
Prompt Engineering Implications: The significant impact of prompt design on model performance has important theoretical and practical implications. Our findings demonstrated that structured prompting approaches (Heuristic and Chain of Thought) consistently improve performance, with average improvements of 4.7% over baseline approaches. However, the effectiveness of specific techniques varies significantly across models and tasks.
The superiority of Heuristic prompts across all models suggests that explicit rule-based guidance helps models activate appropriate reasoning pathways. This finding aligns with theories of prompt engineering that emphasize the importance of providing clear computational frameworks. The success of this approach indicates that models benefit from explicit procedural guidance, even when capable of inferring procedures from examples.
The surprising underperformance of Chain of Thought prompting for Claude-3 challenges assumptions about universal prompting strategies. While CoT improves the performance for GPT-4o and Julius AI on reasoning tasks, it actually decreases Claude-3’s accuracy compared with simpler techniques. This suggests that optimal prompting strategies depend on model architecture and training approaches, requiring empirical validation rather than theoretical assumption.
The high prompt sensitivity observed in Claude-3 (up to 13.4% performance variation) has important implications for deployment. Organizations using this model would need to invest significantly in prompt optimization and maintain prompt libraries for different task types. This additional complexity increases deployment costs and reduces model accessibility for non-expert users.
Error Patterns and Failure Modes: Our detailed error analysis revealed systematic patterns that inform both model improvement and deployment strategies. The distribution of error types provides insights into model limitations and suggests mitigation strategies:
Calculation errors, most prevalent in aggregation tasks, indicate limitations in numerical processing capabilities. These errors often involve incorrect counting, averaging, or statistical computations when processing large datasets. The higher rate of calculation errors in Claude-3 (8.7%) compared with GPT-4o (3.2%) suggests differences in how models handle numerical operations. Potential mitigation strategies include breaking complex calculations into steps or using external tools for numerical processing.
Misinterpretation errors frequently involve ambiguous query terms or complex filtering criteria. These errors highlight the importance of precise task specification and suggest that models struggle with implicit requirements. The lower misinterpretation rates in GPT-4o indicate a superior contextual understanding, possibly due to more comprehensive training data or architectural advantages in handling ambiguity.
Complete failures, while relatively rare, provide important insights into model limitations. These failures typically occur with highly complex reasoning tasks requiring the integration of multiple concepts or extended reasoning chains. The pattern of failures suggests that current models have effective maximum complexity thresholds beyond which performance degrades catastrophically rather than gracefully.
Inconsistency errors, particularly prevalent in Julius AI (4.6%), indicate response stability issues. These errors manifest as different responses to identical queries or semantically equivalent prompts. This inconsistency poses challenges for applications requiring deterministic outputs and suggests the need for response validation mechanisms in production deployments.
Computational Efficiency Considerations: The relationship between performance and computational efficiency revealed important trade-offs for practical deployment. Claude-3’s faster response times (average 2.8 s) come at a significant accuracy cost, while Julius AI’s higher latency (average 4.1 s) does not translate to a proportionally superior performance. GPT-4o’s balanced profile (average 3.2 s with highest accuracy) suggests the effective optimization of the performance–efficiency trade-off.
Response time analysis showed that task complexity significantly impacts computational requirements, with reasoning tasks requiring 2–3× longer processing time than extraction tasks. This scaling pattern has important implications for system design, suggesting the need for differentiated service levels based on task complexity. Applications requiring real-time responses may need to restrict usage to simpler task categories or employ asynchronous processing for complex queries.
The variance in response times also differs across models, with Julius AI showing the highest variability. This unpredictability complicates capacity planning and user experience design. Systems using Julius AI would need larger buffers and more sophisticated queue management to maintain consistent service levels.
Domain-Specificity and Generalization Boundaries:
Critical Assessment of Domain-Specific Findings:
Our evaluation is explicitly limited to structured bibliometric data processing, and readers must exercise caution when generalizing the findings to other domains. Key domain characteristics affecting generalization are as follows:
What transfers: The evaluation framework’s core elements—systematic task categorization by complexity, multi-dimensional performance assessment, and comparative prompt engineering—are domain-agnostic methodological contributions. The finding that task complexity affects all models similarly likely generalizes, as does the importance of prompt engineering.
What may not transfer: Specific performance rankings may differ in other domains. For example:
  • In creative writing tasks requiring linguistic fluency over factual accuracy, performance hierarchies might reverse.
  • In medical diagnosis requiring specialized terminology, all models might perform poorly without domain fine-tuning.
  • In multilingual tasks, models with different language coverage would show different patterns.
  • In reasoning about abstract concepts (philosophical arguments, ethical dilemmas), performance might not degrade systematically with complexity.
Empirical validation is needed for the following:
  • Unstructured text domains (social media, customer reviews, news articles).
  • Creative applications (story generation, poetry, advertising copy).
  • Specialized domains (medical, legal, scientific, technical).
  • Multimodal tasks (image captioning, document understanding).
  • Conversational applications (dialogue, customer service, tutoring).
Framework Extension Roadmap: To extend this framework’s applicability, future research should explore the following:
  • Develop domain-specific task taxonomies maintaining the complexity hierarchy.
  • Create domain-appropriate ground truth through expert annotation.
  • Adapt prompt templates to domain conventions and terminology.
  • Validate that performance metrics remain meaningful in new contexts.
  • Conduct cross-domain meta-analysis to identify universal vs. domain-specific patterns.
Practical Deployment Recommendations: Based on our comprehensive analysis, we offer evidence-based recommendations for model selection and deployment:
It is important to note that our evaluation restricted all models to pure LLM inference without tool use. In production scenarios, Julius AI’s code execution capabilities might substantially improve its performance on aggregation and calculation tasks. Our findings therefore represent lower bounds on Julius AI’s practical utility for analytical applications.
Table 9 provides a quick reference guide for model selection based on application requirements. The following sections elaborate on these recommendations with supporting evidence from our evaluation.
Table 9. Practical model selection guide.
For high-stakes applications requiring maximum accuracy and reliability, GPT-4o represents the optimal choice. Its superior performance across all task categories, combined with low error rates and consistent behavior, justifies potential additional costs. Applications in healthcare, finance, or legal domains would benefit from GPT-4o’s reliability.
For balanced requirements where cost-effectiveness is important, Julius AI provides a competitive performance with potentially favorable licensing terms. However, organizations should account for higher latency and employ appropriate infrastructure to handle response time variability. This model suits applications like content generation, customer support, or educational tools where perfect accuracy is less critical than reasonable performance.
For specialized applications with well-defined task scopes, Claude-3 may provide an adequate performance with faster response times. However, successful deployment requires significant investment in prompt engineering and the acceptance of lower performance on complex tasks. This model might suit high-volume, simple processing tasks where speed is prioritized over accuracy.
Regardless of model choice, our findings emphasize the importance of prompt engineering as a critical competency. Organizations should invest in developing prompt libraries, testing frameworks, and optimization processes. The significant performance improvements from optimized prompts (up to 13.4%) justify this investment.
Limitations and Threats to Validity: While our evaluation provides comprehensive insights, several limitations should be acknowledged:
Methodological Limitations:
  • Domain Specificity: Our evaluation uses bibliometric data exclusively. While the task structure generalizes, performance patterns may differ in other domains (medical, legal, technical).
  • Sample Size: With 15 tasks per category, our statistical power is adequate for detecting large effects (Cohen’s d > 0.8) with adequate power (1 −   β = 0.80), but may be insufficient for detecting smaller effects or for strong generalization claims. Future work should employ larger task sets (n > 50 per category) to enable more robust statistical conclusions about prompt technique effectiveness. However, our effect sizes were consistently large (d > 0.5), suggesting that our main findings regarding prompt sensitivity would likely be replicated with larger samples.
  • Web Interface Dependencies: Our use of web-based interfaces introduces several reproducibility challenges: (1) Hidden system prompts: Web interfaces may include system prompts not visible to users, potentially affecting model behavior. We cannot guarantee that identical prompts submitted via API would produce identical results. (2) Version ambiguity: Models accessed through web interfaces may be updated without user notification, making it difficult to ensure long-term reproducibility. (3) Rate limiting and caching: Web interfaces may employ caching or load balancing that affect response consistency. (4) Interface-specific formatting: Web UIs may preprocess or postprocess text in ways not transparent to users. To partially mitigate these concerns, we (1) documented exact model versions when available, (2) conducted all evaluations within a compressed time window (8 weeks) to minimize version drift, (3) verified response consistency through repeated trials.
  • English Language Only: All evaluations utilized English-language data, limiting the generalizability to multilingual applications.
  • Static Evaluation: Models were evaluated at single time points; performance may evolve with updates.
  • No Tool Augmentation: Evaluation restricted models to pure LLM inference, potentially underestimating the practical capabilities of tool-augmented systems like Julius AI with code execution enabled.
  • Limited Task Diversity: While spanning three complexity levels, our tasks focus on structured data processing and may not capture performance on creative, conversational, or highly specialized tasks.
Implications for Future LLM Development: Our findings suggest several priorities for advancing LLM capabilities:
Improving numerical processing and calculation accuracy remains a critical need, particularly for models targeting analytical applications. Current error rates in aggregation tasks indicate that architectural innovations or hybrid approaches combining symbolic and neural processing might be necessary.
Enhancing consistency and reliability, particularly for complex reasoning tasks, would significantly expand practical applications. Current performance degradation with task complexity limits deployment in scenarios requiring sophisticated analysis or decision-making.
Reducing prompt sensitivity would improve model accessibility and reduce deployment complexity. Models that maintain their performance across diverse prompt formulations would be more practical for organizations with limited ML expertise.
Optimizing the performance–efficiency trade-off remains important for practical deployment. Models that achieve a high accuracy with predictable, reasonable response times would enable broader adoption across resource-constrained environments.
Future Research Directions: Our work identifies several important areas for future investigation:
Priority Future Research Directions:
  • Cross-Domain Validation: Replicate evaluation framework across domains (healthcare, legal, technical, creative writing) to test the generalizability of the findings.
  • Multilingual Assessment: Extend evaluation to non-English languages and cross-lingual tasks to assess language-dependent performance variations.
  • Tool-Augmented Evaluation: Compare pure LLM performance with tool-augmented configurations (code execution, calculators, search) to understand capability boundaries.
  • Longitudinal Tracking: Establish continuous evaluation infrastructure to track performance evolution across model versions and updates.
  • Larger Task Sets: Develop comprehensive benchmark suites with 100+ tasks per category to enable more robust statistical conclusions.
  • Automated Prompt Optimization: Investigate automated prompt engineering techniques to reduce human expertise requirements.
  • Fine-tuning Effects: Examine how domain-specific fine-tuning affects comparative performance and prompt sensitivity.
  • Human–LLM Collaboration: Evaluate hybrid workflows combining human judgment with LLM capabilities for complex reasoning tasks.
  • Emerging Capabilities: Assess multimodal understanding, long-context processing, and emergent capabilities not covered in the current evaluation.

6. Conclusions

This comprehensive comparative analysis of three state-of-the-art Large Language Models provides crucial insights into current LLM capabilities and practical guidance for model selection and deployment. Through systematic evaluation incorporating multiple prompt engineering techniques across diverse task categories, we revealed significant performance variations that have important implications for real-world applications.
Key Findings: Our results demonstrated that GPT-4o currently offers a superior performance across all evaluated dimensions, achieving 89.3% overall accuracy with particularly strong results in data extraction tasks (96.7%). The model’s low error rates (10.7% total), high consistency (91.2%), and balanced response times (average 3.2 s) make it suitable for high-stakes applications requiring reliability. Julius AI provides a competitive performance at 85.7% overall accuracy with consistent results across task categories, though with higher latency (average 4.1 s). Claude-3 showed a more variable performance at 72.1% overall accuracy, with particular challenges in complex reasoning tasks (60.0%) and high prompt sensitivity (standard deviation of 11.3%).
The impact of prompt engineering techniques proved substantial, with structured approaches (Heuristic and Chain of Thought prompts) consistently outperforming simpler formats across most model–task combinations. These findings highlight the critical importance of prompt design in LLM deployment and suggest that prompt engineering should be considered a core competency for organizations implementing AI solutions. The average performance improvement of 4.7% from optimized prompts represents significant practical value, particularly for large-scale deployments.
Performance degradation from extraction to reasoning tasks was observed across all models, with Claude-3 showing the most pronounced decline (26.7 percentage points). This pattern revealed fundamental limitations in current LLM architectures when handling tasks requiring deep semantic understanding and complex inference chains. The consistency of this pattern across different architectural approaches suggests that advances in model design are needed to achieve a human-level performance on sophisticated reasoning tasks.
Research Contributions:
Methodological Contributions:
This work advances the field through several key contributions:
First, we introduce a reusable evaluation framework combining systematic prompt engineering with hierarchical task categorization. This framework enables the comprehensive assessment of model capabilities while maintaining reproducibility and fairness. The three-tier task structure effectively captures increasing cognitive demands and reveals performance patterns not visible through traditional single-task evaluations. The primary lasting contribution of this work extends beyond the specific performance metrics reported for these model versions. Our evaluation framework—combining systematic prompt engineering, hierarchical task categorization, and multi-dimensional performance assessment—provides a reusable methodology applicable to future LLM versions and emerging models. As the field continues to advance rapidly, this framework enables consistent, reproducible evaluation that facilitates meaningful comparisons across model generations.
Second, we provide statistically validated performance comparisons using standardized metrics and rigorous hypothesis testing. Our analysis goes beyond simple accuracy measurements to include response time, consistency, error patterns, and explanation quality. The use of effect sizes alongside significance tests provides practical guidance for interpreting performance differences.
Third, our detailed error analysis reveals systematic failure patterns that inform both model improvement and deployment strategies. By categorizing errors into specific types and analyzing their distribution, we provide insights into model limitations and suggest targeted mitigation approaches.
Fourth, we demonstrate the critical importance of prompt–model interaction effects. Our finding that optimal prompting strategies vary by model challenges assumptions about universal superior practices and emphasizes the need for empirical validation in specific deployment contexts.
Fifth, our work establishes a reproducible baseline for ongoing LLM evaluation. By using publicly available data and detailed methodology documentation, we enable the independent verification and extension of our findings.
Theoretical Contributions:
Beyond the practical findings, this work advances theoretical understanding in three key areas:
Prompt Engineering Theory: Our systematic comparison of five prompting techniques reveals that optimal prompt design is not universal but depends on the interaction between model architecture and task complexity. The finding that Claude-3 performs worse with Chain of Thought prompting on certain tasks challenges the assumption that more explicit reasoning guidance always improves performance. This suggests that prompt engineering theory must account for model-specific response patterns rather than seeking universal prompting strategies. Our work contributes to understanding prompt engineering as a model-dependent optimization problem rather than a one-size-fits-all methodology.
Model Evaluation Methodology: This study advances LLM evaluation methodologies by demonstrating that single-metric assessment is insufficient for understanding model capabilities. Our multi-dimensional framework—combining accuracy, consistency, response time, and error pattern analysis across diverse task complexities—reveals performance characteristics invisible in traditional benchmarks. The systematic variation of both task complexity and prompt design enables the isolation of model-specific vs. task-specific performance factors, advancing more nuanced evaluation approaches.
LLM Capability Boundaries: Our results provide empirical evidence for fundamental capability limitations in current LLM architectures. The consistent performance degradation with task complexity across all models (despite different architectural approaches) suggests shared constraints in transformer-based architectures for complex reasoning. The high calculation error rates in aggregation tasks indicate that current LLMs struggle with precise numerical processing, pointing to architectural needs for hybrid symbolic–neural approaches. These findings help delineate the boundary between tasks that are well suited to pure LLM approaches versus those requiring augmentation with external tools or alternative architectures.
Practical Implications: Our analysis provides actionable guidance for practitioners selecting LLMs for specific applications. Organizations requiring high reliability across diverse tasks should favor GPT-4o, accepting potential higher costs for a superior performance and consistency. The model’s robust performance across different prompt formulations reduces deployment complexity and maintenance overhead. Applications in healthcare, finance, legal analysis, or other high-stakes domains would particularly benefit from GPT-4o’s reliability profile.
For organizations seeking a balanced performance with potentially favorable cost profiles, Julius AI offers a competitive alternative. While showing higher latency and some inconsistency issues, the model provides an adequate performance for many business applications. Content generation, customer support, educational tools, and general business analytics represent suitable use cases where Julius AI’s performance profile aligns with requirements.
Claude-3 may be suitable for specialized applications where its faster response times provide value and task complexity remains limited. However, successful deployment requires significant investment in prompt engineering and acceptance of a lower performance on complex tasks. The high-volume processing of simple queries, basic content classification, or initial screening applications might benefit from Claude-3’s speed advantage.
The high sensitivity to prompt design observed across all models emphasizes the need for systematic prompt development and testing. Organizations should establish prompt libraries, employ A/B testing frameworks, and maintain prompt version control. The significant performance variations based on prompt design (up to 13.4% for Claude-3) justify treating prompt engineering as a critical optimization parameter alongside model selection.
Broader Impact: As Large Language Models continue to transform natural language processing applications across diverse domains, systematic comparative evaluation becomes increasingly critical for realizing their potential while understanding their limitations. This work provides both immediate practical guidance and a methodological foundation for ongoing assessment of these powerful but complex technologies.
The significant performance variations revealed by our analysis emphasize the importance of careful model selection and configuration for specific use cases. As organizations increasingly rely on LLMs for critical applications, understanding these performance characteristics becomes essential for risk management and quality assurance.
Our findings also highlight the continued importance of human expertise in deploying AI systems effectively. The substantial impact of prompt engineering and the need for task-specific optimization demonstrate that successful LLM deployment requires skilled practitioners who understand both model capabilities and application requirements.
Final Remarks: This comprehensive evaluation demonstrates that while current LLMs represent remarkable achievements in natural language processing, significant opportunities remain for improvement. The performance gaps identified, particularly in complex reasoning tasks, indicate that achieving human-level language understanding remains an open challenge. However, the strong performance on extraction and aggregation tasks shows that current models provide substantial value for many practical applications.
As the field continues to advance rapidly, we hope that this work provides a useful framework for ongoing evaluation and comparison. By establishing a rigorous methodology and comprehensive metrics, we contribute to more informed decision-making in model selection and deployment. The insights provided should help practitioners navigate the complex landscape of LLM options while understanding the trade-offs involved in different choices.
The future of natural language processing will likely involve continued model improvements, new architectural innovations, and a superior understanding of how to effectively deploy these powerful tools. Through systematic evaluation and careful analysis, we can guide this development toward systems that are not only more capable but also more reliable, efficient, and accessible for diverse applications and user communities.

Author Contributions

Writing original draft, M.A. and P.M.; supervision, P.M.; writing review and editing, F.S., F.C., P.V. and J.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

Appendix A. Detailed Experimental Results

This Appendix provides comprehensive details of our experimental results, including complete task specifications, extended performance matrices, and additional statistical analyses.

Appendix A.1. Complete Performance Matrix

Table A1 presents the full performance matrix across all combinations of models, task categories, and prompt techniques.
Table A1. Complete performance matrix with confidence intervals.

Appendix A.2. Detailed Prompt Templates

We provide complete prompt templates for all five techniques applied across task categories. Each template maintains semantic equivalence while varying presentation format.

Data Extraction Task Templates

Example Task: Count publications by author “Smith, J.”
  • Prefix: “Count the number of publications authored by ‘Smith, J.’ in the dataset.”
  • Cloze: “The dataset contains ___ publications authored by ‘Smith, J.’”
  • Anticipatory: “To find publications by ‘Smith, J.’: (1) Search the author field for exact matches, (2) Count all matching records, (3) Report the total. What is the count?”
  • Heuristic: “Apply this rule: FOR EACH record, IF author_name EXACTLY_MATCHES ‘Smith, J.’ THEN increment counter. Report final counter value.”
  • Chain of Thought: “Example: To count papers by ‘Doe, J.’, I searched all records and found 3 exact matches. Now, using the same method, count papers by ‘Smith, J.’”

Appendix A.3. Reasoning Task Evaluation Rubric

Table A2 presents the complete scoring rubric for contextual reasoning tasks with criterion weights and scoring guidelines.
Table A2. Complete reasoning task evaluation rubric.
Scoring Formula:
Total   Score   =   ( Correctness   ×   0.35 )   +   ( Clarity   ×   0.20 )   +   ( Detail   ×   0.15 )   +   ( Coherence   ×   0.20 )   +   ( Edge   Cases   ×   0.10 )
where
  • Correctness = score on correctness criterion (0–3).
  • Clarity = score on clarity criterion (0–3).
  • Detail = score on detail level criterion (0–3).
  • Coherence = score on coherence criterion (0–3).
  • Edge Cases = score on edge case handling (0–3).
  • Weights: 0.35, 0.20, 0.15, 0.20, 0.10, respectively.
Final Accuracy Rating:
  • 90–100%: Correct.
  • 70–89%: Partially Correct (counted as 0.5 in accuracy calculations).
  • Below 70%: Incorrect.

Appendix A.4. Statistical Power Analysis

Table A3 presents the post hoc power analysis demonstrating achieved statistical power for major comparisons.
Table A3. Post hoc statistical power analysis.

Appendix A.5. API Validation Study

We conducted a validation study comparing web interface and API results for GPT-4o and Claude-3 on a subset of 20 tasks (n=100 total evaluations). The results showed 94.3% agreement for GPT-4o and 91.7% for Claude-3, with differences primarily in explanation verbosity rather than factual accuracy. This suggests our main findings are robust to interface differences, though some response characteristics may vary.

Appendix A.6. Response Time Percentiles

Table A4 provides detailed response time percentiles across different task categories.
Table A4. Response time percentiles by model and task category (seconds).

Appendix A.7. Inter-Rater Reliability Analysis

Table A5 presents detailed inter-rater reliability statistics for human evaluation components.
Table A5. Inter-rater reliability statistics.

References

  1. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 2020, 33, 1877–1901. [Google Scholar]
  2. OpenAI. GPT-4 Technical Report. arXiv 2023, arXiv:2303.08774. [Google Scholar] [CrossRef] [Scilit]
  3. Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 2023, 55, 195. [Google Scholar] [CrossRef] [Scilit]
  4. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; Bowman, S.R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP; Association for Computational Linguistics: Stroudsburg, PA, USA, 2018; pp. 353–355. [Google Scholar]
  5. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; Bowman, S. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2019; Volume 32. [Google Scholar]
  6. Rogers, A.; Kovaleva, O.; Rumshisky, A. A primer on neural network models for natural language processing. J. Artif. Intell. Res. 2021, 57, 345–420. [Google Scholar]
  7. Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; Zhang, Y.; Narayanan, D.; Wu, Y.; Kumar, A.; et al. Holistic evaluation of language models. arXiv 2022, arXiv:2211.09110. [Google Scholar] [CrossRef] [Scilit]
  8. Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A.A.M.; Abid, A.; Fisch, A.; Brown, A.R.; Santoro, R.; Stuhlmüller, A.; Swayamdipta, A.; et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv 2022, arXiv:2206.04615. [Google Scholar] [CrossRef] [Scilit]
  9. Iorliam, A.; Ande, A.T.; Orban, P.O. A comparative analysis of generative artificial intelligence tools for natural language processing. J. Comput. Sci. Technol. 2024, 24, 78–95. [Google Scholar] [CrossRef] [Scilit]
  10. Upreti, S. Comparative Analysis of Pre-Trained Models for Natural Language Processing. Master’s Thesis, Mälardalen University, Västerås, Sweden, 2023. [Google Scholar]
  11. Zheng, L.; Chiang, W.L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv 2023, arXiv:2306.05685. [Google Scholar]
  12. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q.V.; Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 2022, 35, 24824–24837. [Google Scholar]
  13. Li, X.L.; Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Online, 1–6 August 2021; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 4582–4597. [Google Scholar]
  14. Arora, S.; Narayan, A.; Chen, M.F.; Orr, L.; Guha, N.; Bhatia, K.; Chami, I.; Sala, F.; Ré, C. Ask me anything: Simple strategy for prompting language models. In Proceedings of the International Conference on Learning Representations, Kigali, Rwanda, 1–5 May 2023. [Google Scholar]
  15. Hanafi, A.; Suriani, M.; Ahmad, R. A comparative study of AI decision-making systems: Performance analysis and implementation challenges. In Proceedings of the International Conference on Data Science and Analytics, Shenzhen, China, 13–16 October 2022; pp. 156–163. [Google Scholar]
  16. Singhal, K.; Azizi, S.; Tu, T.; Mahdavi, S.S.; Wei, J.; Chung, H.W.; Scales, N.; Tanwani, A.; Cole-Lewis, H.; Pfohl, S.; et al. Large language models encode clinical knowledge. Nature 2023, 620, 172–180. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  17. Katz, D.M.; Bommarito, M.J.; Gao, S.; Arredondo, P. GPT-4 passes the bar exam. Philos. Trans. R. Soc. A 2024, 382, 20230254. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  18. Tschoetsch, T. Comparison of language models: Performance, efficiency, and practical considerations. AI Res. Q. 2023, 8, 45–62. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.