GenAI-Powered Text Personalization: Natural Language Processing Validation of Adaptation Capabilities

Abstract

The authors conducted two experiments to assess the alignment between Generative AI (GenAI) text personalization and hypothetical readers’ profiles. In Experiment 1, four LLMs (i.e., Claude 3.5 Sonnet, Llama, Gemini Pro 1.5, and ChatGPT 4) were prompted to tailor 10 science texts (i.e., biology, chemistry, and physics) to accommodate four different profiles varying in knowledge, reading skills, and learning goals. Natural Language Processing (NLP) was leveraged to evaluate the GenAI-adapted texts using an array of linguistic and semantic features empirically associated with text readability. NLP analyses revealed variations in the degree to which the LLMs successfully adjusted linguistic features to suit reader profiles. Most notably, NLP highlighted inconsistent alignment between potential reader abilities and text complexity. The results pointed toward the need to augment the AI prompts using personification, chain-of-thought, and documents regarding text comprehension, text readability, and individual differences (i.e., leveraging RAG). The resulting text modifications in Experiment 2 were better aligned with readers’ profiles. Augmented prompts resulted in LLM modifications with more appropriate cohesion features tailored to high- and low-knowledge readers for optimal comprehension. This study demonstrates how LLMs can be prompted to modify text and uniquely demonstrates the application of NLP to evaluate theory-driven content personalization using GenAI. NLP offers an efficient, real-time solution to validate personalized content across multiple domains and contexts.

1. Introduction

Reading is a fundamental learning activity that supports students in acquiring new knowledge and developing expertise across domains [1]. To build a coherent understanding of complex topics, readers must integrate information from texts into an interconnected mental model [2]. While learning from texts is essential for academic success, it can be challenging for students who lack prior knowledge of the topic or sufficient reading skills to understand the content [3,4,5]. Furthermore, reading comprehension is influenced by both reader characteristics and textual features. Text readability is the extent to which a reader can process and understand a text [6,7]. The same text can be easy or challenging to a reader depending on their reading skills and familiarity with the topic. Given the significance of reading comprehension in learning and the variability in reader characteristics, personalized learning has emerged to address students’ diverse needs [8,9,10].

1.1. Personalized Learning

Personalized learning involves matching students with materials at an appropriate difficulty level, tailoring contents to suit their unique interests, thereby enhancing comprehension and engagement, and supporting vocabulary acquisition, skill development, and overall academic success [11,12]. When students perceive learning materials as relevant and aligned with their interests, they are more likely to persist through challenges and actively engage in learning activities, even when the content is difficult [13,14]. Moreover, students engaged in personalized learning experiences demonstrate improved knowledge retention and higher gains in mathematics and reading performance compared to those in traditional learning settings [15,16]. For example, tailoring algebra problems based on students’ interests has been shown to enhance student’s motivation, engagement, and learning outcomes [17].

Personalized learning has emerged as a strategy to align text readability and writing style appropriate for individual needs [18]. Reading texts suited to students’ abilities also fosters interest, motivation, and a passion for learning [19,20]. Prior research has demonstrated the importance and efficacy of matching text to readers’ abilities to optimize comprehension and learning outcomes [21,22]. To effectively align reading materials with students’ reading capabilities, the Lexile Framework is a widely used text–reader matching tool that determines the reading skills of children and matches them with texts based on a readability formula [23,24]. This framework quantifies text difficulty based on factors such as syntactic complexity, sentence length, and word frequency, while reader skill is assessed using standardized assessments [24,25].

While established frameworks like Lexile [23,24,25] have demonstrated effectiveness, advancements in artificial intelligence (AI) provide promising tools to implement personalized learning at scale. Generative AI (GenAI) plays a crucial role in advancing personalized learning by dynamically tailoring learning materials specifically to students’ needs and abilities. Large Language Model (LLM) applications have been applied to adapt writing style, simplify content, and generate materials tailored to individual needs [26,27]. For instance, in educational contexts, LLMs were leveraged to simplify content and adjust lexical and syntactic complexity to enhance accessibility for students with learning disabilities [28]. Similarly, researchers have leveraged LLMs and OpenAI’s API to automatically generate personalized reading materials and comprehension assessments within the learning management system [29,30,31]. These personalized experiences resulted in higher student engagement and increased their study time, ultimately enhancing academic performance and outcomes [32,33,34].

Although the positive impact of personalization on learners’ engagement and performance is well-established, implementation of GenAI-powered personalization in the classroom remains challenging. While text adaptation using LLMs offers a scalable solution to personalize learning, evaluating the effectiveness of modifications can be complicated. Effective text personalization requires rapid validation and iterative refinements to ensure that materials continuously align with students’ evolving needs and learning goals. Traditional assessment (e.g., human comprehension study) is both time-consuming and resource-intensive, which makes it challenging and impractical to implement on a large scale. To address this challenge, an automated and scalable validation method is necessary to evaluate the extent to which personalized content aligns with students’ various needs and assets.

1.2. Evaluating Text Differences Using Natural Language Processing

Natural Language Processing (NLP) is a computational approach that systematically analyzes and extracts patterns from text [7]. In educational and psychological research, NLP tools have been widely applied to automatically evaluate linguistic and semantic features of texts. NLP analyses are extensively used in computer-based learning systems to evaluate text quality and assess student-generated responses (e.g., self-explanations, think-aloud responses, essays, and open-ended answers) [8]. These linguistic analyses provide insights into task performance, cognitive attributes, and individual differences such as literacy skills and prior knowledge [35,36].

Another application of NLP analyses is text readability assessment. Text readability is the extent to which a reader can process and understand a text and it is quantifiable based on various linguistic indices (e.g., cohesion, noun-to-verb ratio, syntactic, and lexical complexity; [8,37]). NLP tools (e.g., Coh-Metrix; [37]) have been leveraged to extract and analyze features that influence text readability based on discourse processing theories. For instance, cohesion, syntactic complexity, and lexical sophistication are associated with coherence-building processes and reading comprehension for different students [8,37,38]. Additionally, NLP analyses afford systematic distinguishing between different types of texts and discourse based on linguistic and semantic features and, in turn, evaluate the difficulty of educational materials [38,39]. Scientific texts often contain dense information, abstract concepts, and complex sentence structures that are challenging for students to process and understand [40,41]. Moreover, academic vocabulary, which is commonly used in academic text, increases comprehension difficulty because readers need to have relevant background knowledge to infer word meanings [41,42].

Text cohesion (e.g., connectives, causal, referential, and coreference) or how well ideas are connected within a text is another property of text that is measured using NLP. Cohesion influences how readers synthesize information and construct meaning [43]. High-cohesion texts benefit low-knowledge readers who lack the necessary background knowledge to infer meanings due to conceptual gaps [43]. In contrast, low-cohesion texts are more suitable for high-knowledge readers, as they facilitate deep learning by encouraging inference generation using prior knowledge [43].

Comprehension difficulties arise not only from a lack of reading skills or prior knowledge but also from a mismatch between textual features and the individual characteristics of the reader [44,45]. When readers lack requisite background knowledge or reading skills, linguistic features such as complex syntax or low cohesion can exacerbate comprehension challenges [8]. Skilled readers are able to comprehend advanced academic texts because they possess high-quality lexical representations and apply effective reading strategies [44]. These strategies enable them to efficiently decode complex syntactic structures, infer the meaning of unfamiliar vocabulary from context, and integrate prior knowledge to support deeper understanding. In contrast, readers with limited skills or knowledge may struggle with texts that are less cohesive or that contain sophisticated vocabulary and sentence structures [45,46]. Additionally, less skilled readers may have difficulty processing complex syntax, as effective comprehension requires sustained attention and cognitive engagement [47,48].

Table 1. Linguistic features related to text readability. Source: authors’ contribution.

Features	Metrics and Descriptions
Overall Readability	Flesch–Kincaid Grade Level (FKGL): Indicates text difficulty based on sentence length and word length Academic writing: The extent to which the texts include domain-specific words and sophisticated sentence structures, commonly found in academic writing texts Development of ideas: The extent to which ideas and concepts are developed and elaborated throughout a text
Conceptual Density and Cohesion	Noun-to-verb ratio: Text with a high noun-to-verb ratio results in dense information and complex sentences that require greater cognitive effort to process Sentence cohesion: The extent to which the text contains connectives and cohesion cues (e.g., repeating ideas and concepts)
Syntax Complexity	Sentence length: Longer sentences often have more clauses and complex structure Language variety: Indicates the extent to which text varies in the language used (sentence structures and wordings)
Lexical Complexity	Sophisticated wording: Lower measures indicate the vocabulary familiar and common, whereas higher measures indicate more advanced words Academic frequency: Indicates the extent of sophisticated vocabulary are used, which are also common in academic texts

describes several linguistic measures related to text complexity that have been shown to correspond to comprehension difficulties faced by students [37]. We measured these linguistic features using NLP because these linguistic patterns provide objective, quantifiable measures of text difficulty that correlate with expert judgments of readability [49]. To effectively tailor text readability for diverse readers with unique characteristics, it is essential to examine the alignment between linguistic features and reader abilities. Additionally, assessing multiple linguistic features related to text difficulty rather than relying on a single metric provides a more comprehensive assessment of readability [8]. Texts written using academic language, uncommon vocabulary, high lexical density, and complex syntax pose challenges for many readers [50]. In addition, text comprehensibility can be enhanced by revising materials to include clearer causal relationships, explicit explanations, and more elaborated background information for less skilled, low-knowledge readers [43,45].

As adaptive learning technologies evolve, automated methods for quick and efficient assessment of text modifications are essential. Leveraging NLP analyses to validate modifications offers several advantages over traditional human assessment. NLP tools can extract linguistic metrics, quantify text complexity, and provide an objective validation method for text personalization. NLP is efficient and scalable, which allows for iterative evaluation of content across various domains without requiring multiple human studies. NLP-based assessments can be scaled to provide immediate feedback, enabling real-time assessment and text refinement. As a result, personalized content can be modified based on quantitative feedback and assessed and adapted iteratively. Thus, in this study, we leveraged NLP to assess these linguistic features and determine how text adaptations varied in complexity across reader profiles and LLMs. The linguistic features of each text modification were extracted using the Writing Analytics Tool (WAT; [51]). By quantifying linguistic properties using NLP, we aimed to evaluate how LLMs tailored the modifications to accommodate the needs of diverse readers.

1.3. Current Research

In the following experiments, we examined how Generative AI (GenAI) can personalize educational texts for different reader profiles by leveraging NLP tools to validate modifications. In Experiment 1, we prompted LLMs to modify scientific texts to accommodate four different reader profiles varying in knowledge, reading skills, and learning goals. In Experiment 2, we refined prompting strategies to improve text personalization and focused on one LLM to examine the effect. NLP analyses were leveraged to evaluate how different LLMs and prompting techniques adjust text readability to examine the extent to which modifications align appropriately with the unique needs of readers. We aimed to explore two research questions: 1. To what extent do different LLMs adapt scientific texts for different reader profiles? 2. How do augmented prompting strategies influence the quality of modified texts? These studies highlight how NLP analyses can be applied to validate text readability in real time across large datasets. The NLP-based validation method also facilitates rapid assessment and iteration so that adaptive learning technologies can quickly tailor content to suit students’ needs. Rapid testing and refinement help ensure that learning materials are effectively tailored and personalized to optimize student’s learning.

2. Experiment 1: LLM Text Personalization

2.1. Introduction Experiment 1

Due to the inherent differences in model design (i.e., training corpus and tuning strategies), different LLMs may produce varied outputs even when given identical prompts [52]. The corpus that LLM is trained on can have a significant influence on its performance and suitability for different task applications. For instance, an LLM trained with academic texts may perform better in educational or scientific content generation, while a model trained with code may excel in programming-related tasks [52]. Recent research comparing outputs from widely used LLMs such as Claude, Llama, ChatGPT, Gemini, and DeepSeek highlighted noticeable differences in output patterns [53]. Furthermore, different versions of the same model (e.g., GPT-3.5 vs. GPT-4) also show outputs with varied levels of lexical richness and syntactic complexity [54,55,56].

Considering these sources of variability, it is important to examine how different LLMs personalize learning materials. In Experiment 1, we investigated how four commonly used LLMs (Claude 3.5 Sonnet, Llama 3.1, Gemini Pro 1.5, and ChatGPT 4) modify scientific texts to suit distinct reader characteristics and learning contexts (e.g., prior knowledge, reading skills, and learning goals). The goal was to apply NLP analyses to assess the extent to which linguistic features varied based on reader profiles and whether the models differed in their ability to personalize texts. Ten texts from biology, chemistry, and physics were adapted to align with different reader profiles. We used NLP to extract and analyze linguistic features in the modified texts, evaluating the extent to which the outputs were tailored to the intended readers. We hypothesized that readability-related linguistic features would vary across profiles and that each LLM would exhibit unique output patterns even when prompted identically.

2.2. Materials and Method Experiment 1

2.2.1. LLM Selection and Implementation Details

The four LLMs selected for the comparative analysis were Claude 3.5 Sonnet (Anthropic), Llama 3.1 (Meta), Gemini Pro 1.5 (Google), and ChatGPT 4 (OpenAI). See Appendix A for details (e.g., versions, dates of usage, and training size and parameters) of the four LLMs. These models were selected for comparison based on three criteria. First, they are selected because of the comparable training size and parameters, which suggests comparable capabilities in language comprehension and generation. Secondly, these LLMs are easily accessible and commonly used across settings (i.e., personal, academic, and industry). Lastly, all four LLMs have a well-established reputation for high performance for general-purpose and instructional NLP tasks. While the training sizes vary across models, they are all well-known for comparable capabilities in language understanding and generation of contextually relevant text. Each model is known for its proficiency in natural language understanding and generating contextually relevant text.

2.2.2. Reader Profiles Experiment 1

Four hypothetical reader profiles were created to simulate readers’ unique characteristics and assess LLMs’ capability to adapt texts to the persona. The characteristics of these four reader profiles differ across background knowledge and reading levels. Each profile included detailed information on the readers’ age, educational background, reading level, prior knowledge, interests, and reading goals (see

Table 2. Descriptions of various reader profiles provided to the LLMs in Experiment 1. Source: authors’ contribution.

	Descriptions
Reader 1 (High RS/High PK *)	Young professional (25-year-old) who has a strong educational background in science and engineering. Interested in physics High prior knowledge in natural sciences subjects, especially physics Learning goal: To develop an in-depth understanding of scientific concepts and principles to pass a class exam
Reader 2 (High RS/Low PK *)	Young adult (20-year-old) who is a curious learner. Enjoys learning about science in an engaging and accessible way Low background knowledge in natural sciences subjects Learning goal: To develop an understanding of scientific concepts and principles to pass a class exam
Reader 3 (Low RS/High PK *)	College student (18-year-old) struggles with reading because of dyslexia. Feels anxious and frustrated when reading complex scientific texts. Lacks confidence in reading comprehension High background knowledge in natural science subjects Learning goal: To develop an understanding of scientific concepts and principles to pass a class exam
Reader 4 (Low RS/Low PK *)	Student (18-year-old) who is looking for a fun and engaging introduction to scientific concepts. Enjoys imaginative and relatable explanations Limited science knowledge and low understanding of scientific terms. Learning goal: To develop an understanding of scientific concepts and principles

* RS = reading skills; PK = prior knowledge.

for reader profile descriptions). While four profiles are limited to explore the extent to which LLM can tailor text to each reader, we designed these four as a proof-of-concept to initially explore personalized adaptations and NLP capabilities in evaluating personalized content.

2.2.3. Text Corpus

Ten expository texts in the domains of biology, chemistry, and physics were compiled. These texts were selected from the iSTART website www.adaptiveliteracy.com/istart (accessed on 21 August 2024) (The texts are publicly available through the iSTART website (www.adaptiveliteracy.com/istart) (accessed on 21 August 2024). Users may create a free account on the website and navigate to the “Texts Library” menu. These materials are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), (https://creativecommons.org/licenses/by/4.0/) (accessed on 6 June 2025) (e.g., iSTART StairStepper, 55).) [57]. The texts were chosen for their varying levels of complexity and relevance to scientific topics, which are suitable for testing the models’ ability to modify text based on different reader profiles.

Table 3. Scientific texts. Source: authors’ contribution.

Domain	Text Title	Word Count	FKGL *
Biology	Bacteria	468	12.10
Biology	The Cells	426	11.61
Chemistry	Chemistry of Life	436	12.71
Biology	Genetic Equilibrium	441	12.61
Biology	Food Webs	492	12.06
Biology	Patterns of Evolution	341	15.09
Biology	Causes and Effects of Mutations	318	11.35
Physics	What are Gravitational Waves?	359	16.51
Biochemistry	Photosynthesis	427	11.44
Biology	Microbes	407	14.38

* Flesch–Kincaid Grade Level.

provides the list of the texts, domains, titles, number of words, and the Flesch–Kincaid Grade Level.

2.2.4. Procedure Experiment 1

The four LLMs were prompted to modify 10 scientific texts to suit four different reader profiles varying in prior knowledge and reading skills. See Appendix B for details of the prompts used. The LLMs were prompted to modify each text to improve comprehension and engagement for a reader, with the goal of tailoring the text to align with the unique characteristics of the reader (age, educational background, reading skills, prior knowledge, and reading goals). Table 2 provides the descriptions of the four reader profiles that varied as a function of prior knowledge (PK) and reading skills (RS). For each reader profile, the procedure was repeated 10 times, each time with one text. Then, the conversation history was erased, and the same procedure was repeated for the next reader profile, modifying 10 science texts using the same prompt. In total, each LLM was prompted 40 times, generating 160 modifications for four reader profiles from four LLMs (Claude, Llama, ChatGPT, and Gemini).

The linguistic features (as shown in Table 1) were automatically extracted using WAT [51], which provides linguistic and semantic indices associated with writing quality. These indices serve as validation metrics to evaluate linguistic and semantic features of the LLM text personalization. By aligning linguistic features with given reader profiles, we provide a basis to assess the extent to which LLM modifications appropriately tailor texts to suit readers’ needs (e.g., low-knowledge readers benefit from cohesive text and simplified syntax vocabulary; high-knowledge readers benefit from lower cohesion and increased complexity). In

Table 4. Hypothesized linguistic features of adapted texts aligned to reader profiles. Source: authors’ contribution.

Reader Profiles	Linguistic Features Aligned for Reader
	Overall Readability	Conceptual Density and Cohesion	Syntax and Lexical Complexity
Reader 1 (High RS/High PK)	High Flesch–Kincaid Grade Level (FKGL *), high complexity High academic writing	Low cohesion to encourage active engagement High information density	Complex sentence structures, high sentence length Sophisticated vocabulary Varied language and syntax
Reader 2 (High RS/Low PK)	Moderate text complexity Moderate academic writing	Introduce scientific concepts with elaborated explanations and examples Moderate cohesion	Complex sentence structures, moderate sentence length Sophisticated vocabulary Varied language and syntax
Reader 3 (Low RS/High PK)	Moderate text complexity Moderate academic writing	Elaborated explanations of key concepts High cohesion	Simplify sentence structures, moderate sentence length Minimal technical vocabulary Clear and concise language
Reader 4 (Low RS/Low PK)	Low FKGL *, low text complexity High academic writing	Elaborated explanations of key concepts High cohesion	Simplify sentence structures, short sentence length Minimal technical vocabulary Clear and concise language

* Flesch–Kincaid Grade Level.

, we outline how linguistic features should be adapted to align with the characteristics of various reader profiles.

2.3. Results Experiment 1

2.3.1. Main Effect of Reader Profile on Variations in Linguistic Features of Modified Texts

The goal of this analysis was to examine the extent to which the linguistic features of LLM-modified texts aligned with different reader profiles. A 4 × 4 MANCOVA was conducted to examine the effects of reader profiles (High RS/High PK, High RS/Low PK, Low RS/High PK, and Low RS/Low PK) and LLMs (Claude, Llama, Gemini, and ChatGPT) on linguistic features of modifications, with word count included as a covariate. The main effect of reader profiles on linguistic features was significant. See

Table 5. Descriptive statistics and main effects of reader profiles. Source: authors’ contribution.

Linguistic Features	Reader 1 (High RS/High PK *)		Reader 2 (High RS/Low PK *)		Reader 3 (Low RS/ High PK *)		Reader 4 (Low RS/Low PK *)		Main Effects of Profile
	M	SD	M	SD	M	SD	M	SD	F (3, 320)	p	η2
FKGL	16.97	2.36	10.63	1.76	9.84	1.89	8.50	1.39	355.64	<0.001	0.79
Academic Writing	89.93	12.53	45.08	24.86	37.20	23.25	17.17	15.95	228.05	<0.001	0.70
Idea Development	57.38	28.05	47.40	25.37	48.19	23.94	45.19	23.95	4.97	0.002	0.05
Sentence Cohesion	55.00	32.55	50.30	29.19	40.85	23.96	48.81	26.84	2.67	0.04	0.03
Noun-to-Verb Ratio	2.81	0.62	1.93	0.25	1.84	0.31	1.87	0.25	133.37	<0.001	0.58
Sentence Length	20.91	6.59	18.70	4.46	14.75	3.27	16.31	3.64	30.42	<0.001	0.24
Language Variety	75.75	21.26	54.07	21.43	27.14	18.46	33.88	18.85	112.79	<0.001	0.54
Sophisticated Word	90.23	9.97	42.87	19.35	31.17	17.71	23.50	13.59	342.11	<0.001	0.78
Academic Frequency	9591.39	1425.57	8708.02	1328.34	7763.13	1426.14	8016.06	1308.47	30.42	<0.001	0.24

* RS = reading skills; PK = prior knowledge.

for descriptive statistics and F-values of the main effects of reader profiles. Results showed that linguistic features of modifications significantly differed across reader profiles.

As expected, the modifications for Profile 1 (High RS/High PK) had the highest syntactic and lexical complexity measures, followed by Profile 2, then 3 and 4. Texts modified for Profile 1 had significantly higher Flesch–Kincaid Grade Level (FKGL) (M = 16.97, SD = 2.36) compared to texts modified for Profile 2 (High RS/Low PK) (M = 10.63, SD = 1.76), p < 0.001, Profile 3 (Low RS/High PK) (M = 9.84, SD = 1.89), p < 0.001, and Profile 4 (Low RS/Low PK) (M = 8.50, SD = 1.39), p < 0.001. Pairwise comparisons also revealed significant differences across reader profiles for measures of text complexity, including academic writing, noun-to-verb ratio, sentence length, language variety, sophisticated wording, and academic vocabulary, aligning with the hypothesized difficulty levels for each profile. As intended, texts modified for Profile 1 had the highest complex syntax and lexical measures, followed by Profiles 2, 3, and 4 (all p < 0.05). Contrary to expectations, cohesion was higher for Profile 1, which has high knowledge, compared to Profiles 2 and 4, which both have low prior knowledge (p < 0.05).

2.3.2. Main Effect of LLMs

The main effect of the models was significant. Linguistic features of modifications from different LLMs differed regardless of reader characteristics. See

Table 6. Descriptive statistics and main effects of LLMs. Source: authors’ contribution.

Linguistic Features	Claude		Llama		Gemini		ChatGPT		Main Effects of LLMs
	M	SD	M	SD	M	SD	M	SD	F (3, 320)	p	η2
FKGL	11.13	4.34	11.87	3.18	11.23	3.91	11.72	3.53	3.35	0.02	0.03
Academic Writing	44.43	34.11	54.14	32.34	45.47	34.45	45.34	31.29	4.70	0.01	0.05
Idea Development	59.97	22.63	33.77	16.90	51.52	23.93	52.91	30.29	19.58	<0.001	0.17
Sentence Cohesion	30.38	24.15	60.86	24.71	52.65	25.32	51.06	31.06	20.30	<0.001	0.17
Noun-to-Verb Ratio	2.25	0.80	2.11	0.48	2.06	0.43	2.03	0.43	6.27	<0.001	0.06
Sentence Length	14.71	3.91	18.68	5.06	18.55	4.25	18.73	6.22	17.68	<0.001	0.16
Language Variety	47.61	28.43	38.07	27.84	55.51	25.77	49.64	25.68	12.21	<0.001	0.11
Sophisticated Word	46.21	31.90	46.55	26.74	47.58	30.07	47.43	32.51	0.15	0.93	0.00
Academic Frequency	7851.69	1465.06	9420.10	1569.22	8646.06	1291.46	8342.75	1412.02	21.48	<0.001	0.18

for descriptive statistics and F-values.

Llama’s modifications (M = 11.87, SD = 3.18) had significantly higher FKGL compared to modifications by Claude (M = 11.13, SD = 4.34), p < 0.05. Llama’s modifications are significantly higher in academic writing and academic wording compared to modifications by Claude, Gemini, and ChatGPT (all p < 0.05). Modifications by Claude (M = 30.38, SD = 24.15) had significantly lower sentence cohesion compared to modifications by Llama (M = 60.86, SD = 24.71), p < 0.001; Gemini (M = 52.65, SD = 25.32), p < 0.001; and ChatGPT (M = 51.06, SD = 31.06), p < 0.001. Modifications by Gemini had the highest language variety (M = 55.51, SD = 25.77) compared to texts generated by Claude (M = 47.61, SD = 31.9), p < 0.05, and Llama (M = 38.07, SD = 27.84), p < 0.001.

2.4. Discussion Experiment 1

NLP analyses showed variations in linguistic features that were adjusted by various LLMs to address individual differences. These variations ensure that the modifications are comprehensible and suitable for the given reader. Specifically, modifications generated for skilled and high prior knowledge profiles showed greater syntactical and lexical complexity, academic writing style, and sophisticated language. In contrast, texts modified for skilled, low-knowledge profiles featured long sentences, sophisticated wording, and varied language while maintaining moderate information density that can avoid overwhelming readers with many science concepts due to lower knowledge. Modifications for low-knowledge and less skilled readers simplified sentence structures and wording to increase readability. LLMs’ sensitivity to different readers’ needs led to variations in linguistic features of modified texts, which were captured by NLP analyses.

Moreover, each LLM’s modifications differed linguistically regardless of reader profiles. Claude’s modifications contained short sentences and were the least cohesive but also had the highest levels of idea development. Llama’s modifications had the highest complexity measures (e.g., FKGL, academic vocabulary, and academic writing). Gemini balanced the depth of information with engaging and cohesive language and used relatable analogies and illustrative examples, which are particularly beneficial for educational materials. ChatGPT modifications were cohesive and elaborated concepts thoroughly in the modifications, especially for low-knowledge profiles.

However, NLP analyses also revealed that cohesion features were not appropriately aligned with readers’ needs, which contradicts findings for optimal comprehension suggested by prior research [43,45]. These inconsistencies in modification highlighted the need to improve prompting techniques and provide LLMs with more explicit instructions and background information to guide the modification process. The single-shot prompt used in Experiment 1 lacks context-relevant information that could guide LLMs to perform the task more precisely. Having detected limitations in the generic prompt by leveraging NLP measures, we refined and tested an augmented prompt that was more detailed and structured to improve text personalization.

3. Experiment 2: Prompt Refinements

3.1. Introduction Experiment 2

In Experiment 1, the LLMs did not effectively tailor cohesion features to align with the readers’ background knowledge. Specifically, texts modified for high-knowledge readers were highly cohesive compared to those for low-knowledge readers. Cohesion plays a critical role in comprehension and learning because conceptual gaps can promote active processing to generate bridging inferences [43,45]. As a result, readers with low prior knowledge benefit from cohesive texts because they lack the necessary background knowledge to fill in conceptual gaps. In contrast, texts adapted for high-knowledge readers should contain less cohesive cues to foster knowledge-bridging and deep comprehension. The mismatch in cohesion for modifications for high- versus low-knowledge readers suggested that the LLMs require further instruction on how to adjust cohesion appropriately for these profiles. Specifically, there may be some limitations in the LLM’s ability to fully differentiate text cohesion for this specific reader profile, highlighting the need for further refinement in tailoring text modifications for less-skilled, high-knowledge readers.

Findings from NLP analyses also showed that idea development was significantly higher in modifications tailored for high-knowledge and skilled readers compared to modifications for all other profiles. When reading for academic purposes, it is crucial that students can learn effectively from the text. As such, scientific concepts should be sufficiently and equally elaborated for all students and not just the skilled ones. It is critical that text modifications for low-knowledge or less skilled profiles simplify vocabulary and syntax but remain cohesive and comprehendible.

One of the most effective methods to enhance LLM performance is by instructing the models with clear and structured prompts. Providing LLMs with additional context on how to complete the task also helps LLMs to generate more accurate outputs and follow adaptation guidelines suggested by prior research [58,59]. High-quality instructional prompts also help LLMs reduce hallucinations and help models adhere more closely to intended outcomes. Prompting plays a critical role in determining the quality and relevance of a model’s responses [52]. The input prompt can influence the variability in performance such that ambiguous or underspecified prompts can lead to inconsistent outputs [52]. Similarly, prior research has demonstrated that well-structured and carefully designed prompts consistently enhanced task performance across diverse NLP tasks [60]. Prompt quality is a major factor for LLM performance, especially for tasks involving reasoning, adaptation, or following instructions.

These findings underscore the importance of systematic prompt design in enhancing output quality. Based on the findings of Experiments 1 and 2, we identified several areas for improvement from the current prompt to enhance text personalization. Iterative prompt improvement can potentially improve LLM’s capability in tailoring texts to specific reader profiles. As such, the primary focus of Experiment 2 is on enhancing prompts to optimize the LLM performance in text personalization and improve the alignment between LLM-generated personalized texts and specific reader profiles.

Several refinements were made to strengthen the instruction prompt used in Experiment 1. See

Table 7. Modified descriptions of various reader profiles provided to the LLMs in Experiment 2. Source: authors’ contribution.

	Descriptions
Reader 1 (High RS/High PK *)	Age: 25 Educational level: Senior Major: Chemistry (Pre-med) ACT English composite score: 32/36 (performance is in the 96th percentile) ACT Reading composite score: 32/36 (performance is in the 96th percentile) ACT Math composite score: 28/36 (performance is in the 89th percentile) ACT Science composite score: 30/36 (performance is in the 94th percentile) Science background: Completed eight required biology, physics, and chemistry college-level courses (comprehensive academic background in the sciences, covering advanced topics in biology, chemistry, and physics, well-prepared for higher-level scientific learning and analysis) Reading goal: Understand scientific concepts and principles
Reader 2 (High RS/Low PK *)	Age: 20 Educational level: Sophomore Major: Psychology ACT English composite score: 32/36 (performance is in the 96th percentile) ACT Reading composite score: 31/36 (performance is in the 94th percentile) ACT Math composite score: 18/36 (performance is in the 42th percentile) ACT Science composite score: 19/36 (performance is in the 46th percentile) Science background: Completed one high-school-level chemistry course (no advanced science course) Limited exposure and understanding of scientific concepts Interests/Favorite subjects: arts, literature Reading goal: Understand scientific concepts and principles
Reader 3 (Low RS/High PK *)	Age: 20 Educational level: Sophomore Major: Health Science ACT English composite score: 19/36 (performance is in the 44th percentile) ACT Reading composite score: 20/36 (performance is in the 47th percentile) ACT Math composite score: 32/36 (performance is in the 97th percentile) ACT Science composite score: 30/36 (performance is in the 94th percentile) Science background: Completed one physics, one astronomy, and two college-level biology courses (substantial prior knowledge in science, having completed multiple college-level courses across several disciplines, strong foundation in scientific principles and concepts) Reading goal: Understand scientific concepts Reading disability: Dyslexia
Reader 4 (Low RS/Low PK *)	Age: 18 Educational level: Freshman Major: Marketing ACT English composite score: 17/36 (performance is in the 33rd percentile) ACT Reading composite score: 18/36 (performance is in the 36th percentile) ACT Math composite score: 19/36 (performance is in the 48th percentile) ACT Science composite score: 17/36 (performance is in the 34th percentile) Science background: Completed one high-school-level biology course (no advanced science course) Limited exposure and understanding of scientific concepts Reading goal: Understand scientific concepts

* RS = reading skills; PK = prior knowledge.

for changes made to the modified prompts. First, the descriptions of reader profiles lacked specificity and clarity. To address this, we included more detailed and quantifiable metrics to demonstrate reading proficiency and prior knowledge level. Instead of using broad terms like high or low, we predicted that incorporating more specific metrics to describe these attributes would enable the model to better tailor texts. By explicitly including quantifiable measures in the profile description, we aimed to reduce ambiguity and enhance the model’s ability to adjust linguistic features aligned with unique needs. Incorporating objective data about the reader was included to enhance clarity and ensure more consistent and goal-aligned responses. This approach aligns with practices in intelligent tutoring systems (ITSs) in which detailed learner profiles are essential for delivering tailored content to suit each student [61,62,63]. Furthermore, insights from educational assessment and LLM evaluation methodologies suggested that standardized metrics provide a clear context for both human and machine interpretation [64]. As such, in LLM prompt design, the standardized metrics provide a more precise context, which helps LLM tailor the text more effectively and align with the intended complexity level appropriate for the student’s proficiency [59,65].

To quantify reading skills, standardized test scores were used as indicators. For instance, the ACT English composite score (32/36; 96th percentile) and ACT Reading composite score (31/36; 94th percentile) are included to represent a skilled reader. A reader profile with low knowledge of science is represented by ACT Math (18/36; 42nd percentile) and ACT Science percentile ranking (19/36; 46th percentile). The description in the profile includes additional information like “Completed one high-school level biology course, with no advanced science coursework, and limited exposure to scientific concepts”. By including standardized test scores and quantifiable metrics representing reading proficiency and science knowledge, we predicted that LLM performance would be enhanced.

Secondly, we integrated Retrieval-Augmented Generation (RAG) to enhance the contextual relevance of outputs by grounding the LLMs’ responses in empirical findings. Retrieval-Augmented Generation (RAG) is a framework that enhances the accuracy and contextual relevance of output by incorporating an external knowledge base during the text generation process [66,67]. Integrating research articles into the LLMs’ knowledge base helps guide the modification process, which helps the LLMs to align linguistic features to reader abilities [68,69]. The LLM integrates insights by tailoring text features to align with established cognitive science theories on discourse processing without referencing the documents directly.

In our prompt, the LLMs were instructed to retrieve and utilize research papers as guidelines for modifying content. The RAG approach utilized here involves content-grounded referencing in which the retrieved empirical and theoretical documents serve as contextual frameworks to guide the modifications implicitly. The documents guide the adaptation process implicitly rather than through explicit textual referencing. The reference materials provided to the LLMs to support the RAG process consisted of peer-reviewed research papers and book chapters focusing on key domains such as text comprehension, readability, cohesion, and individual differences. We included theoretical papers on comprehension theories, such as the Construction–Integration model [70] and comprehensive model of comprehension [2], which explain how readers construct meaning based on textual features and prior knowledge. Empirical studies were also used to provide grounded evidence on how text cohesion and prior knowledge interact to influence comprehension [43,45]. Moreover, we included research papers on linguistic features of writing quality to guide the adaptation of cohesion, writing style, and lexical and syntactic features. See Appendix D for a complete list of documents.

Two additional techniques were also incorporated into the new prompt: chain-of-thought and personification. Personification involves instructing the LLMs to adopt a role aligned with the task objectives, which leverages its ability to simulate human-like reasoning [59,60]. When researchers prompt LLMs to adopt a specific role (e.g., cognitive scientist), they more effectively leverage the model’s capability to simulate goal-directed behaviors aligned with task expectations [59]. Chain-of-thought involves instructing the model to explain its reasoning process step by step. Prior research has shown that making the reasoning process transparent can enhance LLM performance, accuracy, and consistency because it breaks down tasks into smaller parts, resulting in more coherent and logical responses [71]. Outlining steps and rationales also helps researchers identify errors and refine prompts.

Prompting plays a critical role in the response quality, and making the prompt more detailed and structured enhances task performance. As such, in the second experiment, we used NLP to evaluate the improvement in modification generated using the augmented prompt. By analyzing linguistic features, we assess the alignment between modifications and readers’ unique needs and abilities. The four hypothetical reader profiles in Experiment 1 were modified to be more detailed and structured, following the example in Table 6. The revised prompts were evaluated using the same 10 scientific texts from Experiment 1. Unlike Experiment 1, which compared multiple LLMs, Experiment 2 focused on examining the effect of augmented prompting strategies using Gemini to isolate the variability from other LLMs. The augmented prompts were tested on the same 10 scientific texts, and modifications were analyzed using NLP tools to extract linguistic features such as cohesion, lexical complexity, and sentence structure. The NLP analyses were used to assess whether the augmented prompt led to improvements in text alignment with different reader profiles. By demonstrating how NLP tools can assess suitability between textual features and reader characteristics, this research proposes a scalable solution for evaluating text personalization.

3.2. Materials and Method Experiment 2

3.2.1. LLM Selection Experiment 2

In the second experiment, we refined the prompting process to improve text personalization and focused on the performance of Gemini. Gemini was selected for Experiment 2 because its modifications closely resembled those of ChatGPT in terms of conceptual depth but provided more engaging and cohesive outputs. See Appendix A for details (e.g., versions, dates of usage, and training size and parameters).

3.2.2. Reader Profiles Experiment 2

Four hypothetical reader profiles created for Experiment 1 were modified to improve the structure and level of specificity. These enhancements aimed to provide LLMs with clear and more context about the profiles. For instance, standardized test scores (e.g., ACT composite scores in English, Reading, Math, and Science) were included, which quantified readers’ proficiency levels. Statements about readers’ academic backgrounds and exposure to scientific concepts further clarified each profile’s prior knowledge and topic familiarity. Table 7 provides comprehensive descriptions of these enhanced reader profiles.

3.2.3. Procedure Experiment 2

The goal of Experiment 2 was to enhance prompt clarity and structure to improve the alignment between LLM-generated modifications and the unique needs of diverse reader profiles. Gemini was prompted with the augmented prompt to modify ten scientific texts to suit the four revised reader profiles (see Appendix C). Ten expository texts from Experiment 1 were used for this experiment. See Table 3 for information about the texts. Each generation iteration involved ten modifications for each reader profile. In total, there were 40 modifications for four reader profiles. Each modified text was systematically analyzed using WAT [51] to extract linguistic features related text readability, such as cohesion, lexical complexity, syntax complexity, academic writing style, and idea elaboration. Table 1 describes each linguistic feature included in the analysis. These features provided quantitative validation metrics to assess the extent to which the augmented prompt improved the alignment between text modifications and reader profiles. The linguistic features of modifications generated by Gemini from Experiment 1 were used to compare across the two prompts.

3.3. Results Experiment 2

3.3.1. Academic Writing

To examine the extent to which augmented prompts improved alignment between linguistic features and readers’ needs, a two-way ANOVA (four reader profiles: High PK/High RS, High PK/Low RS, Low PK/High RS, and Low PK/Low RS × two prompt types: single-shot vs. augmented) was conducted. There was a significant two-way interaction effect between reader profile and prompt, suggesting that the impact of the augmented prompt on linguistic features of modified texts varied depending on the reader profiles.

The interaction effects between prompt and profile were significant, F (3, 240) = 10.45, p < 0.001, η² = 0.12. As intended, the results showed that academic writing was significantly lower for texts modified for less skilled and low-knowledge readers using the augmented prompt compared to the single-shot prompt (p < 0.001). See Figure 1 for the main effect of prompt and reader profiles on academic writing.

3.3.2. Conceptual Density and Cohesion

The interaction effect between prompt and profile was significant for cohesion, F (3, 240) = 6.18, p < 0.001, η² = 0.07. As predicted, cohesion was significantly lower for texts modified for skilled and high-knowledge readers using the augmented prompt (p = 0.02) and was significantly higher for less skilled and low-knowledge readers (p < 0.05). See Figure 2 for the main effect of prompt and reader profiles on sentence cohesion.

3.3.3. Syntactic and Lexical Complexity

The interaction effect between prompt and profile was significant for language variety, F (3, 240) = 6.24, p < 0.001, η² = 0.08, and sophisticated wording, F (3, 240) = 17.44, p < 0.001, η² = 0.19. As intended, modifications generated by augmented prompts for high knowledge and skilled profiles significantly increased language variety and sophisticated wording. In contrast, modifications for low-knowledge or less skilled profiles significantly decreased lexical sophistication, aligning with the hypothesized difficulty levels for each profile. See Figure 3 and Figure 4 for the main effect of prompt and reader profiles on language variety and sophisticated wording.

3.4. Discussion Experiment 2

NLP analyses provided quantifiable evidence suggesting that the augmented prompt effectively tailored linguistic features to suit different reader needs. Linguistic features were more appropriately aligned to reader needs based on evidence-based text modifications reported in prior research related to comprehending challenging text. Specifically, texts generated for skilled, high-knowledge reader profiles were more syntactically complex and lexically sophisticated and lower in cohesion. In contrast, the modifications for less skilled and low-knowledge profiles had simpler syntax and more common vocabulary and were more cohesive. Simplifying language and using familiar words improves readability and ease of understanding for less skilled, low-knowledge readers. The results of Experiment 2 suggest that NLP successfully provided objective measures that illustrate how prompt refinements can lead to more effective text personalization.

4. General Discussion

4.1. Text Readability and Reader Alignment

In these two experiments, we leveraged LLMs to modify text for different reader profiles and leveraged linguistic features to evaluate the LLM-generated modifications. Experiment 1 demonstrated that LLMs vary in how they adapt text readability to align with readers’ characteristics, which was evidenced in variations of linguistic features related to text difficulty. In Experiment 2, NLP analyses illustrated how the augmented prompt enhances text personalization. These prompt refinements resulted in better-aligned text readability.

Tailoring text complexity to reader abilities is essential for supporting comprehension and learning, especially for less skilled readers. Students with lower topic knowledge and reading proficiency often struggle with decoding, fluency, and monitoring comprehension [72,73]. These challenges are exacerbated when texts contain dense sentence structures, academic vocabulary, or lack cohesion. Therefore, to support comprehension, personalized texts should simplify syntax (e.g., active voice and shorter sentences) and lexical complexity and use consistent language. Deeper linguistic properties, such as cohesion and conceptual density, also influence comprehension, especially for readers with limited prior knowledge. In our research, personalized texts for low-knowledge readers demonstrated high cohesion through elaborations and connective words. These features have been shown to benefit low-knowledge readers and support comprehension by bridging conceptual gaps [43,45]. In contrast, texts modified for high-knowledge and skilled readers included more sophisticated syntax and vocabulary, more varied language, and a writing style more similar to academic writing. These texts exhibited lower cohesion, which can promote deeper comprehension and learning by encouraging active inference-making [43,45].

4.2. Variability Across LLMs

In addition, NLP analyses revealed variations in linguistic features across LLM outputs, regardless of reader profiles. While all LLMs demonstrated sensitivity to readers’ needs, the modifications varied in writing style and linguistic patterns. Llama produced the most complex texts, marked by extensive use of academic language and sophisticated terminologies. Claude’s outputs were significantly less cohesive compared to other models, while Gemini’s outputs had the highest level of lexical diversity and used varied language structures. LLMs also differed significantly on features such as idea development, sentence length, and academic word frequency, suggesting each model produced outputs with distinct linguistic patterns. These findings aligned with prior research, suggesting that differences in both training data and fine-tuning strategies result in variability in outputs even when prompted with identical input [54,65,74]. It is both practically necessary and theoretically important to examine how LLMs personalize texts when considering applying LLMs to modify learning materials. Even small variations in output can influence readability, coherence, and ease of comprehension, ultimately impacting the quality of personalized learning experiences.

This research further demonstrates the benefits of augmenting information provided to the LLMs using various techniques (e.g., personification, chain-of-thought, and RAG). Generative AI can be leveraged to quickly modify content; there is little doubt that LLMs can generate revised versions of content based on human instructions. However, the dilemma is that there have been few attempts (if any), other than intuition, to validate modifications of content using GenAI. These findings also demonstrated how text analysis using NLP tools can be used to evaluate the linguistic features of LLM-generated texts, providing a basis to assess whether adaptations align with different reader needs. The current research highlighted NLP-based analysis as a practical and scalable validation method for assessing personalization quality. In an adaptive learning system, NLP tools can be leveraged to analyze linguistic features in real time, which supports rapid evaluation and iterative refinement without requiring costly human studies.

4.3. Limitations and Future Directions

In the current research, only four reader profiles were used for generating LLM-adapted texts. While this number provided sufficient proof-of-concept for exploring personalized adaptations and demonstrating the capabilities of NLP analyses in evaluating personalized content, the limited number of profiles potentially constrains the generalizability of our findings. Specifically, the reader profiles might not capture the full complexity and range of individual differences in a diverse student population. Additional characteristics such as interests, motivation, cultural backgrounds, learning disabilities beyond dyslexia, or cognitive characteristics were not sufficiently represented in these profiles [75,76,77]. To accurately personalize and evaluate educational materials, it is essential to capture these broader individual differences beyond reading skills and prior knowledge [78]. Future research should explore more nuanced reader characteristics and incorporate a more diverse set of reader profiles to improve external validity. In addition, it would be beneficial to test on a larger and more diverse text domain.

Another limitation of these two experiments is that each text was prompted to be modified only once for each reader profile. LLMs are inherently non-deterministic, and each iteration can produce varying outputs even when using the same prompts [79]. The variability in responses across identical prompts even within the same model is particularly problematic, especially when they are used for educational tasks that require consistency [80]. As such, conducting single-run trials limits the ability to capture and quantify this internal variability. Future research can systematically analyze repeated text generations to examine reliability and consistency across LLM outputs.

The current research demonstrated how LLMs can be used to generate personalized educational content, as well as how NLP can be leveraged to assess and validate content quality. Notably, however, a persistent challenge is the risk of hallucinations. LLMs inherently generate information that is potentially inaccurate or irrelevant [81,82,83]. While semantic similarity between the original and revised texts was automatically calculated to detect inconsistencies, we still had to have a human in the loop to manually review whether the generated content preserved key concepts and remained relevant to the original text (see Appendix E for the rubric guiding evaluation process). While automating this process would require multi-agent programming in which a separate LLM checked accuracy and relevance, NLP offers an efficient means of supporting quality assessment and iterative refinement of content. This approach is valuable because personalized content can be dynamically and continuously adapted to students’ evolving knowledge, interests, and skills.

5. Conclusions

The novel contribution of this research is the demonstration of an NLP-based validation method that systematically and iteratively assesses GenAI-modified educational texts. Unlike traditional human assessment, which is resource-intensive and time-consuming, leveraging NLP analyses provides a scalable and efficient validation process. Specifically, NLP metrics objectively quantify text complexity and readability, allowing for immediate feedback loops that support rapid iteration and content refinement. In the two experiments, NLP analyses provided objective and quantifiable evidence to evaluate linguistic alignment between LLM-generated modifications and readers’ individual needs. Experiment 1 illustrated that LLMs varied in their capacity to adjust readability appropriately for different reader profiles. Experiment 2 demonstrated how augmented prompting techniques improved alignment with readers’ characteristics, further emphasizing the effectiveness of iterative refinements using NLP-based feedback.

Leveraging NLP to validate LLM modifications has strong potential to support continuous and iterative refinements to enhance personalization. NLP analyses can provide real-time assessment regarding the extent to which content is appropriately tailored to students’ needs and abilities. As a result, it allows for iterative refinements to increase the likelihood that the LLM will generate content aligned with expectations. Modifications can be adjusted based on quantitative analyses and further improved without requiring costly and time-consuming studies with human participants. By quickly tailoring text features with individual needs and validating the content, NLP-based validation approaches have the potential to enhance personalized learning [84,85]. After initial LLM outputs are automatically analyzed, subsequent iterations can integrate NLP-generated feedback. This data-driven iterative cycle allows the system to continuously and dynamically adapt educational content as learners’ skills and knowledge evolve, significantly enhancing personalized learning experiences over time. The current research demonstrated that NLP can serve as a scalable, real-time, and efficient evaluation method for automatically validating LLM-generated personalized content. NLP-based validation can be applied to create adaptive, personalized learning environments that evolve in response to learner growth. Investigating the impact of learner-aligned LLM adaptation on academic performance and engagement represents a critical next step.