Exploring the Application of Large Language Models (LLMs) in Data Structure Instruction: An Empirical Analysis of Student Learning Outcomes in Computer Science

Abstract

Recent advancements in Large Language Models (LLMs), including ChatGPT, DeepSeek, and Claude, have facilitated their growing integration into computer science education, including data structure courses. Despite their widespread adoption, the association between sustained and informal LLM usage and students’ learning outcomes remains insufficiently understood. This study seeks to address this gap by empirically examining the association between LLM usage and undergraduate performance in data structure education. We conduct a twelve-week empirical study involving fifty-four undergraduate students, in which LLMs were made freely accessible but neither explicitly encouraged nor discouraged during coursework and assignments. Students’ LLM usage patterns are analyzed in relation to their academic performance across different task types. Findings reveal a significant negative association between extensive reliance on LLMs for cognitively demanding tasks and overall learning outcomes. Additionally, an inverse associative trend is observed between the frequency of LLM usage across some learning activities and academic performance. In contrast, the use of LLMs for supplementary purposes, including conceptual clarification and theoretical understanding, exhibits a notably positive association with final performance. These findings suggest a task-dependent associative relationship between LLM usage and learning outcomes: LLM usage for conceptual learning shows a positive association with the mastery of relevant knowledge when used as a supplementary learning tool, while excessive LLM usage shows a negative association with the development of fundamental analytical and problem-solving skills. This study highlights the importance of carefully integrating LLMs into data structure education to support learning while preserving students’ independent cognitive engagement.

1. Introduction

In recent years, large language models (LLMs) have been increasingly integrated into artificial intelligence (AI) and computer science education. Transformer-based LLMs like OpenAI’s ChatGPT and Microsoft’s Copilot excel at code generation, concept explanation, and debugging—functions highly relevant to the data structure course. As a foundational computer science course emphasizing logical reasoning and algorithmic design, data structure makes LLMs a promising auxiliary tool for students seeking concept clarification or problem-solving support. While their potential value in data structure education is clear, understanding LLMs’ nuanced impact on learning outcomes remains critical. Previous studies [1,2] explored LLM integration in programming education, noting merits and risks, but most predated the widespread adoption of advanced transformer attention mechanisms. LLM services such as ChatGPT, Claude, Mistral, and Gemini are now readily accessible to non-expert users, including students in data structure courses. Considering their capabilities in explaining abstract concepts, generating algorithmic code, and supporting problem solving, updated research is required to systematically examine the specific functionalities of these advanced LLMs in data structure courses.

However, treating all LLM interactions as a single, uniform risk overlooks the subtle differences in how students actually use these tools. We need to clearly distinguish between using LLMs to generate solutions and using them to clarify concepts. Unlike code generation, using LLMs for conceptual clarification not only helps students eliminate misunderstandings, but also facilitates building a stable thinking pattern without weakening the independent thinking ability. The goal of applying LLMs is not to replace students’ cognitive effort, but to provide support for deep thinking by eliminating external cognitive load caused by obscure terminology or confusion of concepts. Ultimately, this support mechanism can significantly improve learning efficiency and guide students to achieve a deep understanding of professional concepts.

This study examines the impact of informal LLM usage in data structure courses, contributing to broader understanding of LLMs in computer science education. This study aims to explore the relationship between different modes of large language model (LLM) use and undergraduate learning outcomes. It contributes new empirical insights into the role of these tools in supporting the mastery of fundamental concepts, including algorithms, data representation, and problem-solving. To guide this study, we propose the following research questions and hypotheses:

RQ1. 

How does outsourcing problem-solving tasks to LLMs (specifically code generation) correlate with academic performance in data structure courses?

H1. 

Reliance on the code generation capabilities of LLMs exhibits a significant negative correlation with the final performance in data structure courses.

RQ2. 

How does using LLMs for specific tasks, such as generating algorithmic code, seeking conceptual explanations, and debugging data structure implementations, affect students’ final performance?

H2A. 

Frequent use of LLMs for direct code generation is negatively correlated with the final performance.

H2B. 

Appropriate use of LLMs for conceptual analysis is positively correlated with the final performance.

H2C. 

Frequent use of LLMs for error debugging is negatively correlated with the final performance.

These questions and hypotheses guide our investigation into how Large Language Models (LLMs) such as ChatGPT, Claude, DeepSeek, and Gemini influence data structure education. Our goal is to offer concrete, evidence-based insights that can inform teaching practices and support the thoughtful integration of LLMs in data structure courses. Empirical research examining the impact of advanced large language models (LLMs) on data structure education remains limited. Most prior studies are conducted before the widespread availability of high-capability LLMs, offering only partial insights into the educational effects of LLMs integrated with recent AI advancements in this specific course domain.

To address this gap, the present study provides an empirical analysis of how diverse usage patterns of modern LLMs—ranging from algorithm design and code generation to concept explanation and error debugging. Through a controlled experimental design and rigorous statistical analysis, this study advances our understanding of the role of LLMs in basic computer science education and provides targeted insights to support teaching strategies that balance the use of effective LLMs with the development of essential logical reasoning and algorithmic problem-solving skills.

2. Related Work

Educational research has increasingly turned its attention to large language models (LLMs) in light of the widespread availability of advanced, cloud-hosted systems such as ChatGPT, Claude, Qwen, DeepSeek, and Gemini [1,2]. Unlike earlier AI-assisted learning tools that required specialized infrastructure or institutional deployment, these models are freely accessible and widely adopted by students in everyday learning scenarios [3,4]. As such, recent research increasingly focuses on off-the-shelf LLMs that realistically reflect students’ actual learning environments. In line with this trend, the present review concentrates on the informal use of contemporary LLMs in computer science education, with particular emphasis on data structure courses, while excluding self-hosted systems and pre-transformer models that no longer represent current student experiences.

A substantial body of recent work reports the pedagogical potential of advanced LLMs in supporting computer science learning. Prior studies demonstrate that LLMs can assist students through step-by-step explanations of algorithms, real-time debugging feedback, adaptive problem generation, pseudocode refinement, and interactive dialogue-based tutoring [5,6,7,8,9,10]. Empirical evidence further suggests that such affordances can enhance learners’ comprehension of abstract and cognitively demanding topics, including recursion, linked list manipulation, tree traversal algorithms, graph search strategies, and hash table operations [11,12,13,14]. These findings position LLMs as promising auxiliary tools, particularly for lowering entry barriers and providing personalized learning support in foundational computer science courses. Despite these well-documented pedagogical benefits, LLM usage in education still presents notable challenges and inherent risks that require further empirical investigation. One recurring concern is the high sensitivity of LLM outputs to prompt formulation: ambiguous or poorly specified prompts often lead to logically flawed, inefficient, or incomplete solutions, especially in edge cases and time-complexity analysis [15,16,17,18,19]. For novice learners, such errors are difficult to detect, potentially reinforcing misconceptions rather than correcting them. Moreover, excessive reliance on AI-generated code may encourage surface-level copying, reduce active cognitive engagement, and weaken the development of independent problem-solving and debugging skills [20,21,22]. Although more recent models demonstrate improved reasoning capabilities compared to earlier systems such as GPT-3.5 [23], they remain prone to occasional logical inconsistencies and oversimplified explanations.

To address these limitations, several scholars advocate for a pedagogical shift that frames LLMs not as autonomous tutors but as scaffolding tools embedded within structured instructional designs. Recommended strategies include redesigning assessments to emphasize process-oriented evaluation (e.g., oral defenses, handwritten diagrams, or algorithmic flowcharts), incorporating AI literacy into curricula, training instructors to recognize and guide appropriate LLM usage, and cultivating students’ critical evaluation skills when interacting with AI-generated outputs [24,25]. Collectively, these studies suggest that LLMs are most effective when used by learners with sufficient conceptual grounding and under clear instructional guidance. While the existing literature offers valuable insights into the general advantages, limitations, and ethical concerns of LLMs in programming education, several important gaps remain. First, many prior studies rely on self-reported perceptions or short-term interventions, rather than controlled empirical designs that directly link LLM usage patterns to measurable learning outcomes [26,27,28,29,30]. Second, relatively little work distinguishes between different modes of LLM usage, such as code generation, conceptual explanation, and debugging—and their potentially divergent effects on learning. Third, focused investigations into data structure courses, which demand a high level of algorithmic reasoning and abstraction, remain scarce.

To clarify how LLMs relate to student performance, we must move beyond the general “benefit and risk” framework. Instead, we examine these tools through the lenses of Cognitive Load Theory (CLT) and social constructivism. These theories suggest that the effectiveness of artificial intelligence assistance depends on the difference between acquiring declarative knowledge (concepts) and developing procedural skills (coding). From a constructivist perspective, an LLM acts as a “knowledgeable other” within the zone of proximal development, clarifying abstract terminology. This reduces irrelevant cognitive load and frees working memory for relevant processing. Consequently, this promotes the construction of robust mental models, which supports hypothesis H2B regarding the positive association with concept-centered use. Conversely, the acquisition of programming skills relies on procedural knowledge and the resolution of “ideal difficulties”, in which case cognitive struggle is crucial for skill internalization. In this case, there is a risk of cognitive offloading when using LLM to generate code, which bypasses the critical problem-solving process required for building procedural knowledge. The elimination of this necessary cognitive involvement provides a theoretical basis for the negative correlation predicted by code generation usage (H2A). Therefore, educational theory predicts a dichotomy: LLMs act as facilitators when used for conceptual frameworks, but become barriers when used to substitute procedural practice.

In this context, this study contributes to the literature by conducting a targeted empirical analysis of how the informal use of widely accessible LLMs influences undergraduate learning outcomes in data structure education. By examining task-specific usage patterns and their relationship with students’ independent performance under controlled conditions, this work moves beyond theoretical or anecdotal discussions. Focusing on real-world usage of contemporary models—including Qwen, Claude, Gemini, and DeepSeek—it addresses a critical gap in current scholarship and provides evidence-based insights for responsibly integrating LLMs into foundational computer science curricula, balancing technological support with the cultivation of essential cognitive and problem-solving skills.

3. Materials and Methods

This study involved fifty-four second-year undergraduate students and examined the effects of informal large language model (LLM) use on academic outcomes in data structure education. The selection of second-year undergraduates is deliberate and aligned with the research objectives. Participants have completed a first-year introductory programming course covering fundamental concepts such as variables, control structures, basic algorithms, and elementary data types in Python or Java, providing them with essential programming literacy. However, they have not yet taken a dedicated course on data structures and thus lacked formal exposure to core topics such as linked lists, trees, hash tables, or graph algorithms. This intermediate knowledge state enables them to have the ability to solve structured problems, but their understanding of abstract concepts is still relatively preliminary, making them ideal participants in studying how the use of LLMs (e.g., Qwen, Claude, Gemini, and DeepSeek) in controlled educational environments affects the mastery of abstract concepts and problem-solving skills.

In light of the growing adoption of large language models (LLMs) in computer science education, the initial phase of this study allows students to use any LLM tools of their choice, including Qwen, Claude, Gemini, and DeepSeek, for completing data structure assignments. This design aimed to reflect real-world learning and development contexts where students increasingly rely on AI assistants to support algorithm design, code debugging, and conceptual clarification. To evaluate the impact of LLM usage on learning outcomes, we subsequently introduced a controlled phase in which LLM access is explicitly prohibited. This LLM access restriction in the controlled phase enabled us to isolate the effect of LLMs during specific problem-solving tasks. By systematically comparing students’ performance, solution quality, and self-reported understanding in unrestricted and restricted stages, we aim to observe to what extent LLMs have improved or may hinder the educational effectiveness of mastering data structure concepts. Figure 1 presents the design of our experiments.

Figure 1. Overview of the whole experiment.

3.1. Initial Phase

During the initial ten-week phase, students completed four progressively structured assignments that focused on implementing and applying core data structures in Python 3.2. These data structures included linked lists, stacks, queues, binary trees, and hash tables. The assignments emphasized algorithmic reasoning, runtime analysis, and code quality, using object-oriented design and type annotations. The complexity of assignments increased with time. Students started with basic linear structures and advanced to more challenging topics, including recursive tree traversals, hash collision resolution, and comparative analysis of structural trade-offs. Each assignment included real-world scenarios—such as modeling browser history with a stack, task scheduling using a queue, or creating a symbol table with a hash map—to help students understand abstract concepts through practical problem-solving.

As shown in Figure 2, the assignments increased in complexity, starting with basic linear structures and progressing to more advanced topics such as recursive tree traversals, hash collision resolution, and a comparative analysis of structural trade-offs. Each assignment was designed around real-world scenarios, such as modeling browser history with a stack, task scheduling with a queue, and creating a symbol table with a hash map. These practical applications helped ground abstract concepts in a real-world problem-solving context.

Figure 2. 10–week assignments.

Across these assignments, participants were permitted to use LLMs informally, such as debugging logic errors, clarifying abstract data structure concepts, generating pseudocode or test cases, or any other tasks they find helpful. Additionally, they received support from two experienced teaching assistants who offered timely guidance and conceptual feedback as needed. This informal LLM usage, complemented by expert human assistance, closely mirrors real-world learning scenarios in which students increasingly integrate AI-powered tools with instructor or peer support to deepen their understanding of data structures. Figure 3 shows an example of prompt templates used in this study.

Figure 3. Prompt templates.

3.2. Controlled Phase

After completing the above ten-week assignments, the study enters its controlled phase during weeks eleven and twelve. Participants are assigned a carefully designed data structures project in which the use of LLMs is expressly prohibited. However, consistent with standard academic practice in the data structure course, students are permitted to consult official documentation and trusted educational resources (e.g., course notes or algorithm textbooks) to support their work. Within a two-hour time limit, they are required to implement core data structure operations—such as traversing a binary tree, resolving hash collisions, or manipulating linked lists—using only the knowledge they have acquired. The task deliberately avoided introducing new concepts; instead, it assessed participants’ ability to independently apply previously mastered material under constrained conditions, thereby establishing a fair baseline for evaluating the influence of prior LLM usage on autonomous problem-solving performance.

Throughout the assignment, two teaching assistants are responsible for supervising the course to ensure compliance with the no-LLM policy. Their supervision ensures experimental integrity by preventing any unauthorized access to LLM-based tools, whether via web browsers or mobile devices. After completing the study, participants are given a voluntary questionnaire aimed at capturing their learning behaviors and implementation strategies. The questionnaire specifically inquires about how they utilized large language models (LLMs) for tasks such as generating code skeletons, debugging logic errors, explaining recursion, and verifying time complexity. To encourage honest reporting, students are assured that their final grades have already been finalized before questionnaire distribution, eliminating concerns about academic consequences. While participation was optional, students were informed that non-participation would result in exclusion from the subsequent data analysis, thus preserving both ethical autonomy and dataset coherence for robust empirical evaluation.

The assignment administered in the controlled phase serves as the final performance assessment and requires students to independently implement core data structure operations in Python without any language model assistance. It includes two equally weighted components (50 points each, total 100 points): Conceptual and Algorithmic Understanding (theoretical component) and Critical Algorithm Implementation (practical component) (see Figure 4).

Figure 4. Assignment design of the control phase.

The theoretical component focuses on assessing students’ grasp of fundamental concepts and algorithmic logic, including analyzing the time/space complexity of data structure operations, explaining the principles of core algorithms (e.g., recursive tree traversal, hash collision resolution), and identifying edge cases in algorithm design. The practical component requires students to complete three core tasks: (1) design a linked list with insertion, deletion, and traversal functions; (2) build a binary search tree supporting recursive search and in-order traversal; (3) implement a hash table using chaining for collision resolution. For the practical component, students are required to apply object-oriented principles, include type hints for code clarity, and write test cases to verify correctness; modular design, incremental debugging, and thoughtful data representation planning are also evaluated. This assessment was jointly graded by two experienced teaching assistants (with 3+ years of data structure teaching and grading experience) and the course instructor. To ensure marking consistency, all graders participated in a calibration session, where they jointly reviewed three sample assignments to align their understanding of the scoring criteria. Detailed scoring rubrics for both components are provided in Appendix B for transparency and reproducibility.

3.3. Questionnaire Design

The questionnaire is designed in Chinese to align with the linguistic preferences of participants, and its core items are constructed based on the three core learning tasks of the data structure course (code generation, conceptual understanding, and error debugging), which are the most common LLM usage scenarios for students in practical learning. The measurement of LLM usage adopts a 8-point Likert scale (1 = Never to 8 = Almost every time) for the dimensions of code generation, error debugging, and concept analysis. A translated excerpt is provided here for clarity, focusing on large language models (LLMs) usage (Table 1). The complete version of the questionnaire is included in Appendix A for the convenience of replication and evaluation by subsequent researchers.

Table 1. Questionnaire of LLM Usage.

No.	Questions	Response Options
1	I used LLMs for generating code.	____(1–8).
2	I used LLMs for debugging logic errors.	____(1–8).
3	I used LLMs for understanding algorithms and concepts.	____(1–8).

The items in Table 1 capture the key patterns of LLM usage by participants in data structure learning, illustrating how these tools supported conceptual learning, algorithm design, and code implementation. These questions constitute only a subset of a broader questionnaire intended to capture participants’ experiences with LLMs during the implementation phase.

3.4. Measurement of LLM Usage

We quantified students’ LLM usage behavior based on the self-reported questionnaire data, with the Likert scale scores of the three core items as the primary metrics of LLM usage frequency for code generation, concept analysis, and error debugging, respectively. The “number of LLM invocations” mentioned in the subsequent results section refers to the subjective estimated values of students based on their actual usage during the 10-week unrestricted learning phase, reflecting the relative frequency of their LLM usage rather than objective real-time call records.

To verify the reliability of the questionnaire (the internal consistency of the measurement dimensions), we conducted a Cronbach’s $\alpha$ reliability analysis on the three core measurement items. The results showed that the overall Cronbach’s $\alpha$ coefficient of the questionnaire was 0.82, and the Cronbach’s $\alpha$ coefficients of the code generation, concept analysis, and error debugging dimensions were 0.78, 0.80, and 0.85, respectively. All coefficients were greater than 0.7, indicating that the questionnaire had good internal consistency and reliable measurement results, which could effectively reflect students’ actual LLM usage behavior in the data structure course.

4. Experimental Results

4.1. Data Overview

We provide a summary of the volume of code associated with the tasks that the participants completed during this study. The Line of Code (LOC) for each assignment is estimated based on carefully designed reference implementations to accurately reflect the course learning objectives and technical requirements. Specifically, Assignment 1 consists of approximately 152 lines, Assignment 2 consists of approximately 384 lines, Assignment 3 consists of approximately 476 lines, and Assignment 4 consists of approximately 893 lines. These values indicate the volume of code required for each assignment, thereby reflecting differences in complexity and scope across assignments.

In addition, this section outlines the statistical data of the measurement variables used in the analysis. Table 2 and Figure 5 jointly illustrate the extent to which students used LLMs across different aspects of their learning in the above assignments.

Table 2. Descriptive statistics of LLM usage.

Stage	Mean	Median	Standard Deviation	Min	Max
Code Generation	4.39	4.00	1.68	1.00	8.00
Concept Analysis	3.02	3.00	1.25	1.00	8.00
Error Debugging	4.91	4.50	2.08	1.00	8.00

Figure 5. The usage of LLMs in different stages.

As illustrated in Figure 5, students report an average frequency of 4.39 for using LLMs to generate code, with a median of 4.00. Comparatively, error debugging shows a higher mean frequency (4.91), indicating it is the most common usage scenario. The standard deviation of 1.68 indicates significant individual differences in the frequency of students using LLMs to generate code. The average frequency for seeking concept analysis was 3.02 (median = 3.00), a moderate level of usage that reflects students’ selective use of LLMs for clarifying abstract concepts in data structure courses. Meanwhile, the usage of error debugging has a mean of 4.91 and a median of 4.50. These statistical data indicate that students are more inclined to rely on LLMs to assist in debugging code errors. The corresponding standard deviation (i.e., 2.08) indicates that a large proportion of students rely on LLMs to solve code and logic error debugging in course learning. Obviously, we can observe that students rely on LLMs for assistance in key processes of course learning, especially in error debugging and concept analysis. Moreover, students vary individually in their degree of reliance on LLMs, with some exhibiting a relatively high level of dependence during the learning process.

In addition to the analysis of LLM usage mentioned above, we conducted tests on students’ course learning outcomes in weeks 11 and 12 without LLMs. Specifically, the course assessment primarily consists of two parts: conceptual and algorithmic understanding, and the implementation of critical algorithms in code. The total score of the test is 100, with understanding of concepts and algorithms, and implementation of critical algorithms, each accounting for half of the score. Table 3 shows the detailed distribution of test scores. From Table 3, it can be seen that students performed poorly in the test, with a failure rate of 14.8% and relatively few high-scoring students.

Table 3. Distribution of Test Scores.

Score	[0, 60)	[60, 70)	[70, 80)	[80, 90)	[90, 100)
Number of students	8	15	16	13	2

To study the relationship between the test score of the students and their frequency of using LLMs in early assignments, we annotate the distribution of student scores corresponding to different LLM usage frequencies in Figure 6.

Figure 6. The relationship between test scores and the usage of LLMs.

Figure 6a reveals a clear negative trend: as the frequency of LLM invocations for code generation increases, test scores decline significantly. In particular, after more than 4 LLMs invocations, the proportion of low-scoring students increases significantly, indicating that excessive reliance on LLMs to generate code may weaken independent programming ability. As for the concept analysis (i.e., Figure 6b), test performance generally improves with increased LLM usage. The proportion of high scores increased notably as invocations increased from 1 to 4. This indicates that using LLMs to assist in understanding abstract concepts and critical algorithms can effectively improve students’ knowledge mastery level. As presented in Figure 6c, the more frequently LLMs are invoked, the lower the overall test score, and there are more low scores when LLM is called frequently, reflecting that frequent reliance on LLMs for error debugging may weaken students’ ability to independently identify and correct errors.

4.2. Statistical Analysis

From a statistical perspective, this analysis reveals the correlation between the frequency of students using LLMs in different assignments and their final academic performance. This study used the Spearman correlation coefficient ($\rho$) to evaluate the strength of these relationships. To be specific, Spearman’s rank correlation coefficient (denoted $\rho$ or $r_s$) is a non-parametric measure of the monotonic association between two variables. It is computed as the Pearson correlation coefficient applied to the rank-transformed data, making it robust to non-normality, nonlinearity, and outliers. Given two variables $X=(x_1,x_2,\dots ,x_n)$ and $Y=(y_1,y_2,\dots ,y_n)$, we can calculate the correlation coefficient $\rho$ as follows.

• Assign ranks to the values in X and Y, resulting in rank vectors $R_X$ and $R_Y$.
• Compute the Pearson correlation between $R_X$ and $R_Y$:

  $$\rho ={\displaystyle \frac{\mathrm{cov}\left(R_X,R_Y\right)}{{\sigma }_{R_X}{\sigma }_{R_Y}}}={\displaystyle \frac{{\sum }_{i=1}^n\left(R_{X_i}-{\overline{R}}_X\right)\left(R_{Y_i}-{\overline{R}}_Y\right)}{\sqrt{{\sum }_{i=1}^n{\left(R_{X_i}-{\overline{R}}_X\right)}^2}\sqrt{{\sum }_{i=1}^n{\left(R_{Y_i}-{\overline{R}}_Y\right)}^2}}}$$

  (1)

  Given that ranks are integers from 1 to n (or averaged ranks), their means are ${\overline{R}}_X={\overline{R}}_Y={\displaystyle \frac{n+1}{2}}$.

Alternatively, when there are no tied values, a simplified formula can be used: $\rho =1-{\displaystyle \frac{6{\sum }_{i=1}^n{d}_i^2}{n\left(n^2-1\right)}}$. To ensure the robustness and reproducibility of our findings, all correlation analyses were conducted using Spearman’s rank correlation coefficient ($\rho$). Given the non-normal distribution of usage metrics, we employed a commonly used resampling method in educational empirical research to estimate Bootstrap confidence intervals (CIs). Besides, we utilized the Bias-Corrected and Accelerated (BCa) method for interval calculation, which provides better coverage accuracy for skewed distributions.

The significance of these correlations is determined by p-value, which is computed via the exact permutation test. If $p>0.05$, it indicates that the calculated $\rho$ has no statistical significance. In addition, this study adopts the Bootstrap method to compute confidence intervals for each Spearman’s correlation coefficient $\rho$. Given the original sample $\mathcal{D}=\{x_1,x_2,\dots ,x_n\}$, we explain the construction process of a confidence interval for $\widehat{\theta }=s\left(\mathcal{D}\right)$ (e.g., Spearman’s $\rho$) as follows.

• Resampling: Randomly draw n observations with replacement from the original sample to form a bootstrap sample ${\mathcal{D}}^{*b}$ (b = 1, 2,… β).
• Compute the statistic: For each bootstrap sample, calculate the statistic ${\widehat{\theta }}^{*b}=s\left({\mathcal{D}}^{*b}\right)$.
• Repeat: Repeat the above procedure $\beta$ times to obtain the bootstrap distribution $\{{\widehat{\theta }}^{*1},{\widehat{\theta }}^{*2},\dots ,{\widehat{\theta }}^{*\beta }\}$.
• Construct confidence intervals: Sort the bootstrap statistics and take $\alpha /2$ and 1$-\alpha /2$ quantiles as the confidence limits. For instance, the 95% confidence interval ($\alpha =0.05$) is $\left[{\widehat{\theta }}_{\left(0.025\right)}^{*},{\widehat{\theta }}_{\left(0.975\right)}^{*}\right]$.

Bootstrap confidence intervals are employed to assess the precision of the estimates and to indicate that the observed relationships are robust to random sampling variation. The Spearman’s correlation calculation results and the corresponding confidence intervals are shown in Table 4.

Table 4. Spearman’s correlation coefficient.

LLM Usage	Spearman’s Rho	Bootstrap CI	p
Generating Code	−0.571	[−0.680, −0.453]	0.019
Concept Analysis	0.301	[0.023, 0.416]	0.046
Error Debugging	−0.235	[−0.326, −0.129]	0.032

The statistical analysis provides empirical support for the proposed hypotheses based on self-reported usage data, revealing a clear task-dependent trend between students’ use of large language models (LLMs) and their final performance in data structure courses.

First, H1 is strongly supported. The frequency of LLM usage for code generation exhibits a significant negative correlation with final grades (Spearman’s $\rho$ = −0.571, p = 0.019). The corresponding bootstrap confidence interval (CI: [−0.680, −0.453]) does not include zero, indicating that greater reliance on LLMs is associated with poorer academic performance. Second, the effects of LLM usage vary substantially across different assignment types. H2A is confirmed, showing that reliance on LLMs for code generation in high-cognitive-demand tasks, such as algorithm design, is significantly and negatively correlated with student performance ($\rho$ = −0.571, p = 0.019, 95% CI: [−0.680, −0.453]). This finding reflects a significant negative association between excessive dependence on LLM-generated code and students’ ability to independently implement algorithms. In contrast, H2B is supported, as the use of LLMs for conceptual analysis demonstrates a significant positive association with final grades ($\rho$ = 0.301, p = 0.046, CI: [0.023, 0.416]), indicating that LLMs can effectively facilitate conceptual understanding when used as a supplementary learning aid. Finally, H2C is supported, LLM usage for error debugging shows a negative correlation with final performance ($\rho$ = −0.235, p = 0.032, CI: [−0.326, −0.129]). This negative associative trend may reflect an unobserved relationship: students who rely heavily on LLMs for debugging may have fewer opportunities to practice independent error identification and analysis, and this potential pattern requires further causal research to verify. These results emphasize the importance of cultivating independent debugging skills, as excessive reliance on LLMs during debugging may weaken students’ problem-identification and correction abilities.

In summary, the association between LLMs usage and learning outcomes depends critically on how they are used. Excessive reliance on LLMs for core practical assignments, such as code generation and error debugging shows a significant negative association with learning achievement. By contrast, the use of LLMs for auxiliary purposes such as concept analysis shows a significant positive association with learning outcomes. In teaching practice, teachers should guide students to use LLMs as a “scaffold” to support concept acquisition, rather than as a “crutch” that substitutes for independent thinking and coding.

5. Mitigation Strategies: Prompt Training and Usage Policies

Given the negative relationship we observed between excessive reliance on LLMs and performance on cognitively demanding tasks, a critical question emerges: can pedagogical interventions reduce these risks? We argue that structured prompt training and guided usage policies can effectively turn LLMs from potential crutches into supportive learning scaffolds.

First, structured prompt engineering training can reshape how students interact with LLMs, shifting them from passive code copiers to active cognitive engagers. Rather than letting students use vague prompts that produce full, ready-to-use code solutions, such as ’Write a function to sort this list’, instructors can teach students to frame prompts that encourage active cognitive engagement. For example, strategies like ’Explain the logic step by step without providing the final code’ or ’Generate pseudocode first, then evaluate its efficiency’ push students to process the LLM’s output thoughtfully instead of copying it passively. Recent work indicates that such process-oriented prompting maintains the cognitive effort necessary for deep learning while still taking advantage of the LLM’s explanatory capabilities. By teaching students to craft targeted, constraint-based prompts, we can position LLMs as interactive tutors that support reasoning, rather than automated tools that replace it. Second, implementing guided LLM usage policies is critical for establishing clear boundaries. Our findings suggest that neither unrestricted access nor complete prohibition is ideal. Instead, courses can adopt nuanced policies that explicitly distinguish between appropriate and inappropriate LLM usage. For instance, LLMs might be allowed for clarifying concepts, debugging specific errors, or generating test cases, but prohibited from writing core algorithmic logic from scratch. Additionally, asking students to submit reflection logs that document how they used the LLMs and what they learned from their responses can strengthen metacognition and accountability. These policies emphasize that the goal of programming coursework is skill development, not just task completion, and reduce the incentive to outsource high-level thinking.

In summary, while our study emphasizes the risks of unregulated LLM usage in complex programming tasks, it also points to a constructive path forward. The negative associations we observed likely reflect how students currently use these tools, rather than inherent technological limitations. Future educational frameworks should prioritize AI literacy, especially effective and ethical prompting, and establish clear, evidence-based usage guidelines. Subsequent empirical research should directly test whether these structured training programs and policy interventions improve learning outcomes in data structures and related courses.

6. Discussion

6.1. Task-Dependent Associative Relationship of LLMs on Learning Outcomes

This study provides empirical evidence that the impact of LLMs on data structures education is profoundly task-dependent. Our findings confirm a significant negative association between extensive reliance on LLMs for high cognitive demand activities (algorithm design and debugging) and student performance. This negative association may reflect a potential pattern in learning behaviors: students who outsource core problem-solving assignments to LLMs show lower levels of performance in independent skill assessments, and this pattern does not represent a causal relationship (e.g., reverse causality or confounding variables may exist). Conversely, using LLMs for supplementary purposes, such as seeking conceptual clarification, showed a markedly positive association with final performance, indicating their potential as an auxiliary learning aid in the data structure course.

6.2. Caution in Interpreting Causality: Reverse Causality Considerations

However, it is critical to interpret the observed negative correlation between code-generation usage and performance with great caution, especially when inferring causality. While our findings indicate that excessive reliance on LLMs for core coding tasks hinders skill development, an alternative explanation involving reverse causality must be considered. It is plausible that students with weaker foundational programming skills or lower self-efficacy are more likely to resort to LLMs for code generation when confronted with cognitively challenging assignments. In this case, frequent LLM usage may be a symptom of struggling learners seeking support, rather than the direct cause of their poor performance. These students may lack the confidence or ability to engage in independent problem-solving, leading them to rely on LLMs as their primary problem-solving tool. Consequently, the negative correlation likely reflects a complex interplay: while over-reliance on LLMs limits the cognitive effort essential for learning, the tendency to heavily rely on LLMs may be driven by pre-existing ability gaps. Future longitudinal studies that control for baseline student ability should be fully considered.

6.3. Teaching Insights: Guiding Effective LLMs Integration in Education

These results emphasize a critical teaching insight. The primary risk of LLMs integration lies not in the technology itself but in how it is employed. When LLMs are used as shortcuts to bypass independent reasoning (rather than as scaffolding to support students), their usage shows a negative association with learning outcomes. Meanwhile, when LLM is used as an interactive learning aid framework to enhance understanding of theories and concepts, it can significantly improve learning outcomes. Therefore, teachers should move beyond the binary view of LLM usage and instead develop clear guidelines for the use of LLMs. Specifically, teachers should encourage students to use large language models (LLMs) as auxiliary tools to support conceptual understanding, while preventing students from overly relying on LLMs in core practical tasks such as independent thinking and problem-solving. Future work should focus on exploring effective teaching strategies while fully leveraging the advantages of LLMs explanation and guidance, and effectively maintaining and systematically cultivating students’ ability to explore and solve problems independently.

7. Limitations of This Study

7.1. Reliance on Retrospective Self-Reported Data

A primary limitation of this study is its reliance on retrospective self-reported data to measure LLM usage frequency. Administering a single questionnaire after a ten-week interval inevitably introduces recall bias, as students may struggle to accurately reconstruct their past interactions. To mitigate this, our survey instrument focused on estimating ‘typical weekly usage’ rather than requesting exact cumulative counts. This methodological approach shown to enhance estimation accuracy in educational research [30,31,32,33]. Nevertheless, we acknowledge that subjective reporting cannot eliminate measurement error. Unlike studies leveraging real-time interaction logs or weekly journals, our approach lacks the granularity to capture moment-to-moment fluctuations in AI adoption. Consequently, the observed relationships between usage patterns and performance should be interpreted as robust trends rather than precise causal metrics. However, the strong alignment between our empirical findings and theoretical predictions derived from Cognitive Load Theory suggests that, despite potential noise, our instrument effectively captured the fundamental direction and magnitude of the impact of LLM usage.

7.2. Contextual Differences Between Assessment and Professional Practice

It is important to interpret our findings in the context of our assessment design, which was similar to traditional exams: LLMs were either restricted or used secretly. This likely made the negative link between heavy LLM usage and poor performance even stronger for students with high LLM reliance. In exams where LLMs are banned or limited, these students often struggle because their usual workflow does not fit the rules; they waste valuable time creating prompts, checking LLMs outputs, and manually adjusting LLM-generated code (without copy-paste), leading to worse results than peers who use their own knowledge. However, these strict lab-like rules are very different from real professional practice today. In software engineering, LLMs are commonly used as legitimate “copilots” in development, where the goal is efficient problem-solving through human–LLM collaboration. Using LLMs well here boosts productivity rather than hindering it, which stands in contrast to the negative outcomes observed in our restricted exam setting.

7.3. Tool Heterogeneity in LLMs Selection

We should also acknowledge a limitation related to tool heterogeneity. In this study, we let students choose freely from various LLMs such as Claude, Gemini, and DeepSeek. We did this to maximize ecological validity and reflect the authentic “bring-your-own-AI” learning environments students actually experience. But this design also introduces variability in model capabilities. Differences in coding accuracy and how thoroughly each model explains concepts may have added noise to our data. This could have influenced how strong the observed effects appeared. Even so, the strong statistical relationships we identified suggest that the way students interact with LLMs, whether for code generation or seeking explanations, remains a critical factor in predicting learning outcomes, no matter which specific tool they use. Future controlled experiments will isolate this variable by requiring all participants to use the same model. This will let us precisely measure the impact of model capability separately from the effects of usage patterns.

7.4. Absence of Confounding Variable Controls and Baseline Measurements

It is important to acknowledge that this study did not include quantitative measures for students’ prior programming ability, learning effort, or self-efficacy, nor did it employ multivariate control models. While these factors limit our ability to fully isolate the independent effect of LLM usage from potential confounding variables (e.g., students with weaker foundations may rely more heavily on LLMs), the findings provide robust evidence of a significant association between usage patterns and learning outcomes. Consequently, our results should be interpreted as correlational rather than causal. Future research would benefit from incorporating baseline ability assessments and multivariate regression models to further explore the specific nature of this relationship.

7.5. Rethinking LLM-Associated Assessments for Aligning with Industrial Practice

The negative link we found does not mean LLMs are inherently flawed, but rather points to a critical mismatch: testing students by banning or limiting LLMs (contrary to real jobs) penalizes those learning to use new technologies and fails to distinguish strategic LLM usage from outsourcing thinking entirely. This suggests the performance drop may stem partly from the assessment method, not LLMs themselves. This highlights the need for aligned assessment reforms that match real-world software engineering practice, where LLMs are used as legitimate collaborative tools. Future assessments should move beyond banning LLMs to measuring students’ ability to review, refine, and apply LLM-generated solutions in realistic programming scenarios.

8. Conclusions

8.1. Summary of Insights

This study empirically investigates the relationship between the use of LLMs and the academic outcomes of undergraduate students in data structure courses. Our findings reveal that the relationship between LLM usage and learning outcomes in data structures education is fundamentally task-dependent. While this study identifies significant associations rather than causal effects, distinct patterns emerge across different usage types. Extensive reliance on LLMs for high-cognitive-demand tasks, such as algorithm design and debugging, correlates negatively with academic performance. In contrast, using LLMs for conceptual clarification correlates positively with learning outcomes. These associations suggest that the issue lies not with the technology itself but with how it is employed. Specifically, inappropriate usage patterns that bypass core skill development appear to be linked to poorer results. This relationship is likely influenced by confounding variables, including prior programming proficiency and learning self-efficacy. Ultimately, the critical factor distinguishing divergent learning outcomes is whether students utilize LLMs as intelligent tutors to seek explanations or as substitutes to generate solutions. The distinction between using LLM as a “tutor” (seeking explanations) versus a “substitute” (generating solutions) emerges as the pivotal factor determining whether LLMs can promote the learning outcomes of computer science courses.

8.2. Practical Recommendations

We propose three actionable strategies as follows:

• Educational institutions should reform the traditional homework model and shift towards diversified assessments guided by the learning process. Specifically, teachers should focus on examining whether students can understand and interpret the internal logic of the Large Language Models (LLMs) generated code, identify and correct its code defects. Academic ability assessment should no longer focus solely on students’ ability to write code independently. Instead, it should place greater emphasis on their comprehensive capacity to critically evaluate LLMs outputs and systematically optimize AI-generated content.
• Educational institutions should establish and implement an explanation-first LLM interaction model, in which students are required to request conceptual explanations and debugging guidance from LLMs before seeking any code-related outputs.
• Teachers need to develop clear usage guidelines to help students form effective LLM usage patterns. These evidence-based LLM usage patterns improve students’ learning outcomes while ensuring mastery of basic skills and knowledge.

Although this study is limited by the potential biases inherent in self-reported data and the heterogeneity of LLMs tools employed, the findings of this study still emphasize the urgent need for further research in the future. Such studies should incorporate real-time LLM interaction logging to systematically trace the progression of LLM reliance among learners and identify its causal relationship with learning outcomes. Future experimental designs will isolate and control for distinct prompt engineering strategies to identify optimal human–LLM interaction modes that maximize learning outcomes while minimizing overreliance on LLMs.

Appendix A.1. Instructions

Please reflect on your usage of Large Language Models (e.g., ChatGPT, GitHub Copilot, Claude, Bing Chat) during the Data Structures course in this semester. For each statement below, indicate how frequently you engaged in the described behavior.

Appendix A.2. Response Scale

• 1 = Never (I did not do this at all for any Data Structures course tasks).
• 2 = Extremely rarely (Only 1 tentative use during the whole semester, with no actual learning support effect).
• 3 = Very rarely (Cumulative 2–3 uses during the semester, only for individual simple problems such as basic concept queries).
• 4 = Rarely (On average, 1 use every 3 weeks, only for solving single, low-difficulty tasks such as simple syntax debugging).
• 5 = Moderately (On average, 1 use every 2 weeks, for some regular learning tasks such as concept sorting for ordinary assignments).
• 6 = Relatively frequently (On average, 1 use per week, for core links of most assignments such as the generation of initial algorithm code drafts).
• 7 = Frequently (On average, 2–3 uses per week, relying on LLMs to complete most key assignment tasks such as complex debugging and complete code writing).
• 8 = Almost every time (Rely on LLMs for all Data Structures learning and assignment tasks, unable to complete independently without LLMs assistance).

Table A1. Frequency of Specific LLM Usage Behaviors.

Item	Questions	Response
Q1	I used LLMs to generate code (e.g., complete code for algorithm implementation, data structure definition) when finishing data structure assignments.	____(1–8)
Q2	I used LLMs for conceptual analysis (e.g., understanding abstract concepts of linked lists/trees/hash tables, algorithmic logic principles).	____(1–8)
Q3	I used LLMs for error debugging (e.g., troubleshooting code syntax errors, logical errors, runtime exceptions) when finishing data structure assignments.	____(1–8)

Table A1. Frequency of Specific LLM Usage Behaviors.

Item	Questions	Response
Q1	I used LLMs to generate code (e.g., complete code for algorithm implementation, data structure definition) when finishing data structure assignments.	____(1–8)
Q2	I used LLMs for conceptual analysis (e.g., understanding abstract concepts of linked lists/trees/hash tables, algorithmic logic principles).	____(1–8)
Q3	I used LLMs for error debugging (e.g., troubleshooting code syntax errors, logical errors, runtime exceptions) when finishing data structure assignments.	____(1–8)