AcademiCraft: Transforming Writing Assistance for English for Academic Purposes with Multi-Agent System Innovations

Abstract

In the realm of academic English writing, both native (L1) and non-native (L2) speakers face significant challenges due to the complex linguistic structures and conventions required. Existing writing assistance tools, while useful for grammar correction and text enhancement, often fall short by providing only corrected output without explanations, and they typically operate as opaque, proprietary systems. This study introduces AcademiCraft, a novel EAP writing assistance built upon a multi-agent system (MAS) and advanced large-language models (LLMs), integrating state-of-the-art natural language processing (NLP) and linguistic research. AcademiCraft distinguishes itself by correcting and refining academic texts and offering detailed explanations for its revisions, thereby deepening the user’s understanding of writing conventions. The tool’s open and replicable framework contrasts sharply with the closed nature of most commercial products. Through a series of rigorous scientific benchmarks, AcademiCraft has been shown to outperform leading commercial writing assistance tools in several key areas, including its ability to handle complex syntactic structures, improve coherence and cohesion in academic texts, and provide contextually accurate word choices, all while maintaining high levels of grammatical precision. This paper details the methodologies and underlying principles of AcademiCraft, and presents a comparative analysis of its performance, demonstrating its superior capability in supporting both L1 and L2 writers in mastering the complexities of academic English writing.

1. Introduction

English for Academic Purposes (EAP) writing presents a formidable challenge, demanding a high degree of linguistic proficiency and academic rigor. This challenge is not only prevalent among L2 English speakers but also affects L1 writers who must navigate complex linguistic phenomena [1] to produce clear, coherent, and academically appropriate texts. The complexities inherent in EAP stem from its strict adherence to grammatical norms, precise vocabulary usage, and the construction of logically structured arguments, which are essential for academic communication and often pose significant hurdles in learning and mastering academic writing skills.

In response to these challenges, various EAP writing assistance tools have been developed to aid writers in improving their text quality. These tools typically focus on grammatical error correction (GEC) and text polishing to enhance readability and adherence to academic standards. However, while these tools automate the correction process, they often fall short in several key areas. Most notably, current EAP writing assistance systems tend to deliver corrected outputs without providing detailed explanations for the revisions made. This lack of transparency can hinder writers’ linguistic development, as understanding the rationale behind specific corrections is crucial for improving writing skills.

Moreover, the proprietary nature of the technology behind most commercial EAP writing tools obscures their operational principles, which limits users’ and researchers’ ability to assess and learn from the technology. Without access to the underlying algorithms and methodologies, it is difficult for users to critically evaluate the suggestions made by these tools or for educational technologists to integrate these tools effectively into broader educational contexts.

Recognizing these gaps, our research incorporates the latest in NLP and integrates substantial linguistic insights to develop AcademiCraft, an innovative EAP writing assistance. What sets AcademiCraft apart is its dual focus—not only does it provide essential GEC and text enhancements, but it also offers detailed explanations for each revision. This functionality is crafted to deepen the users’ understanding of common writing errors and the most effective correction strategies, thus aligning with the educational objectives of EAP assistance.

By marrying NLP technology with thorough linguistic analysis, AcademiCraft strives to be a transparent, educational tool that does more than just improve text quality—it aims to enhance the writer’s skill and understanding. This approach serves the dual purpose of producing superior academic texts and fostering essential writing skills for sustained academic achievement. In the subsequent sections, we will explore in detail the methodologies, operational principles, strengths, and limitations of AcademiCraft, showcasing how it supports both L1 and L2 writers in mastering the complexities of academic English writing.

The primary contributions of this work are threefold:

• We introduce AcademiCraft, a novel MAS-based EAP writing assistance that not only corrects and enhances texts but also provides detailed explanations for revisions, fostering a deeper understanding of academic writing conventions.
• We develop an innovative approach that integrates LLM-based agents for different aspects of text revision, including grammar correction, sentence-level enhancement, paragraph coherence, and chapter-level organization, providing comprehensive writing support.
• Through extensive empirical evaluation, we demonstrate AcademiCraft’s superior performance compared to leading commercial tools across multiple metrics, including grammatical accuracy, text coherence, and academic language use.

2. Related Work

2.1. EAP

EAP is a specialized branch of English language education aimed at preparing learners for the linguistic demands of higher education and professional academic settings. EAP programs are designed to develop critical language skills tailored to the specific needs of academic contexts, focusing not only on general language proficiency but also on the mastery of academic conventions and genres [2].

In EAP, academic writing is a central skill [3] that encompasses understanding and producing various types of academic texts, such as research papers, theses, dissertations, and conference presentations. These texts demand a high level of formality, precision, and adherence to the conventions of academic discourse. The challenges in EAP writing can be extensive, ranging from structuring arguments coherently to employing appropriate academic tone and style [4,5,6,7].

For L1 speakers, the primary challenge often involves honing their ability to convey complex ideas succinctly and formally [8], transitioning from everyday communicative styles to the specialized registers of academic English. They must learn to deploy advanced syntactic structures [9] and sophisticated vocabulary [10] that accurately convey nuanced arguments and evidence critical engagement with scholarly materials.

For L2 speakers, EAP writing introduces additional layers of complexity. L2 learners must overcome linguistic barriers that include mastering a new vocabulary set, often filled with abstract and discipline-specific terms, and grappling with syntactic structures that are unfamiliar or markedly different from those in their native language [11]. Issues such as verb tense consistency [12], the use of articles, and preposition usage can significantly affect the clarity and professionalism of their academic writing [13].

Furthermore, L2 learners are required to understand and apply rhetorical strategies that may not align with those from their cultural or linguistic backgrounds [14]. This includes developing a sense of audience awareness [15,16,17] and adopting an argumentative structure that supports the norms of English academic writing, which typically emphasizes a clear thesis, logical progression of ideas, and a strong conclusion [18,19].

Overall, mastering EAP writing is a demanding task that necessitates a deep understanding of linguistic, cultural, and disciplinary norms, alongside a continuous refinement of language skills to meet the exacting standards of academic communication.

2.2. EAP Writing Assistance

In the field of EAP, various commercial products have been developed to assist learners in improving their academic writing. This section provides an overview of several notable tools designed to aid users in refining their writing through advanced linguistic technologies.

Pitaya (https://www.mypitaya.com (accessed on 1 July 2024)) offers a feature known as “Rewrite”, which enables users to input academic texts and receive alternative versions with enhanced clarity and style. The tool uses advanced algorithms to restructure sentences and suggest more academically appropriate vocabulary, thereby improving the overall readability and formal tone required in academic writing.

Wordvice AI (https://wordvice.ai/tools/paraphrasing (accessed on 1 July 2024)) features an “AI Paraphraser” designed to help writers rephrase their sentences without losing the original meaning. This tool is particularly useful for academics looking to avoid plagiarism by expressing ideas in new ways, ensuring that their writing maintains a high level of originality and adheres to academic integrity standards.

QuillBot (https://quillbot.com/paraphrasing-tool (accessed on 1 July 2024)) includes a “Paraphrasing Tool” that assists users in rewriting texts to achieve better clarity, variety, and academic tone. By providing synonyms and altering sentence structures, this tool helps users refine their drafts and enhance the persuasive power of their arguments.

Effidit (https://effidit.qq.com/demo (accessed on 1 July 2024)) [20] provides a “Text Polishing” service, which includes a Rewriting feature that focuses on enhancing the linguistic quality of academic texts. This service aims to improve the coherence and cohesion of academic writing, making it more understandable and appropriate for scholarly communications.

DeepL Write (https://www.deepl.com/en/write (accessed on 1 July 2024)) is a comprehensive tool that supports various aspects of writing improvement. It offers suggestions for grammar corrections, style enhancements, and vocabulary substitutions, making it a versatile assistance for non-native speakers striving to meet the rigorous demands of academic writing in English.

Common Challenges

Despite the utility of these tools, there are common issues that limit their effectiveness. Firstly, these products typically provide users with revised results without explaining the rationale behind these changes. This lack of transparency can be problematic for learners who wish to understand the principles of effective academic writing deeply. Just as [21] points out, in the context of tasks like GEC in EAP writing aids, it is crucial to provide users with reasons for the revisions, not merely the revised outcomes. Secondly, the proprietary nature of these technologies poses a significant barrier to academic research and improvement. Since the algorithms and methodologies are not openly shared, it is challenging for researchers to analyze, critique, or enhance these tools, which stifles innovation and limits educational progress in EAP writing assistance.

2.3. Prompt Engineering

Prompt engineering is a crucial technique in the development and utilization of language models, particularly for tasks related to EAP writing assistance. It involves the design and refinement of input prompts to guide language models in generating desired outputs effectively. The importance of prompt engineering lies in its ability to influence the behavior of language models, ensuring that the generated text meets specific criteria and quality standards. Formally, prompt engineering can be defined as follows:

$$\mathrm{Prompt}\mathrm{engineering}=arg\underset{P}{max}\sum _{i=1}^n{\mathbb{E}}_M\left[Q\left(O_i\right|P,C_i)\right]$$

(1)

where P represents the prompt, $O_i$ represents the output, $C_i$ represents the context, and Q represents the quality of the output as determined by the language model M.

The development of prompt engineering has evolved significantly with the advancement of LLMs. Initially, prompts were simple and often manually crafted. However, as the complexity of LLMs increased, so did the sophistication of prompt engineering techniques. Modern approaches utilize various strategies, including evolutionary algorithms [22,23,24] and reinforcement learning [25,26,27,28], to optimize prompts for specific tasks.

In the context of English GEC, prompt engineering plays a pivotal role. EAP writing heavily relies on precise grammar usage, making GEC an essential component. Studies such as [29,30] have demonstrated the effectiveness of prompt engineering in enhancing the performance of LLMs for GEC tasks. Ref. [31] also highlights the critical impact of prompt design on the quality of grammar corrections provided by language models. Furthermore, research by [32] underscores the significant influence of prompt engineering on GEC outcomes, emphasizing its importance in achieving high-quality grammatical corrections in EAP writing.

Prompt engineering is also applied in sentence-level revisions (SentRev) [33], which is another crucial aspect of EAP writing assistance. SentRev aims to elevate subpar academic sentences to publishable standards, addressing issues related to clarity, coherence, and formal tone. According to [33], prompt engineering significantly impacts the feedback quality provided by LLMs for SentRev tasks. The study reveals that even when prompts convey the same meaning, the quality of the feedback can vary greatly across different languages. Moreover, within the same language, different prompt designs can lead to substantial variations in feedback quality, highlighting the nuanced nature of prompt engineering in EAP applications.

2.4. AI Agent

AI agents are autonomous entities that observe and act upon an environment to achieve specific goals. These agents are programmed with algorithms that allow them to perceive their surroundings, make decisions, and execute actions. AI agents can be broadly classified into reactive agents, which respond to stimuli without internal state management, and cognitive agents, which maintain an internal state and utilize past experiences to inform current decisions. Within the field of NLP, AI agents have been applied to various tasks, including machine translation [34], sentiment analysis [35], and text summarization [36,37].

A significant advancement in AI agent technology is the development of MAS. These systems involve multiple interacting agents, each with distinct capabilities and objectives, working collaboratively or competitively to achieve complex goals. The superiority of MAS lies in its ability to handle tasks that are too complex for a single agent by distributing the workload and leveraging the diverse strengths of individual agents.

In the context of NLP, MAS can provide enhanced problem-solving capabilities through distributed processing, parallel execution of tasks, and dynamic adaptation to changing environments. For instance, in machine translation, multiple agents can focus on different linguistic components such as syntax, semantics, and context, thereby improving the overall translation quality [38]. Similarly, in sentiment analysis, different agents can analyze various aspects of a text, including emotional tone, intent, and contextual nuances, resulting in more accurate interpretations [39].

The potential of multi-agent systems in EAP writing assistance is particularly promising. EAP writing involves various complex tasks, such as grammar checking, style enhancement, coherence improvement, and content validation. By employing a multi-agent approach, these systems can offer more comprehensive and nuanced assistance. Each agent within the system can specialize in a specific aspect of writing, providing targeted support that addresses the diverse needs of EAP learners.

Furthermore, multi-agent systems can dynamically interact to provide holistic feedback, addressing multiple facets of writing simultaneously [40]. This adaptability is crucial in EAP contexts, where the requirements and standards can vary significantly across disciplines and educational levels. The ability of multi-agent systems to learn and evolve based on user interactions and feedback further enhances their potential [41], making them an invaluable tool in improving the academic writing process for both non-native and native English speakers.

3. Objectives

The objective of this study is to develop an innovative EAP writing assistance system, AcademiCraft, using multi-agent system (MAS) technology. This assistance aims to enhance the quality of EAP drafts, elevating them to, or beyond, the standard of existing commercial EAP writing tools. Additionally, it seeks to provide users with explanations for suggested revisions, addressing complex linguistic phenomena that typically challenge such enhancement.

AcademiCraft contributes to the field of EAP writing assistance systems through its multi-level granular analysis approach, operating at sentence, paragraph, and chapter levels. This hierarchical framework ensures comprehensive coverage of both linguistic and structural elements essential to academic writing. Unlike traditional tools applying universal enhancement methods, our system recognizes the distinct requirements of different textual components.

The specialized agent architecture of AcademiCraft employs nine dedicated agents, each addressing specific aspects of EAP writing. This division creates a system capable of handling complex linguistic phenomena through collaborative intelligence, thereby overcoming limitations commonly found in monolithic approaches used by commercial tools. The collaborative nature of these agents allows for sophisticated analysis that considers multiple linguistic dimensions simultaneously.

A primary objective of AcademiCraft involves providing transparent explanations for each suggested revision. While current commercial tools frequently operate as “black boxes”, our system enhances user understanding through detailed justifications based on grammar correction rationales, Academic Phrasebank applications, use of hedges and boosters, passive voice implementation, paragraph structure analysis, and move analysis for chapter-level organization. This transparency supports the pedagogical value of the system.

AcademiCraft specifically targets academic writing conventions by incorporating established linguistic frameworks such as lexical bundles, the Academic Phrasebank, and Swales’ CARS model. This focus differentiates our system from general-purpose writing tools and addresses the specific needs of academic writers. Furthermore, we aim to develop a system that exceeds the performance of existing commercial tools in accuracy of grammar correction, appropriateness of academic language suggestions, structural improvements at all text organization levels, and user comprehension of suggested changes.

Current commercial EAP writing tools often treat writing assistance systems as a sequence-to-sequence task, relying heavily on large parallel training datasets. This approach reflects a common critique of traditional deep learning methods—developers struggle to explain the reasoning behind their models’ outputs. Existing tools typically cannot adequately explain suggested revisions, have limited coverage of complex linguistic phenomena specific to academic writing, inadequately consider hierarchical text structures, and insufficiently integrate established academic writing frameworks.

Recent advancements in MAS technology offer new potential for breakthroughs in this domain, prompting our exploration of its capabilities for transforming EAP writing assistance systems. By leveraging the collaborative intelligence of multiple specialized agents, AcademiCraft aims to overcome these limitations and provide a more effective, transparent, and educational writing assistance system experience tailored specifically for academic contexts.

4. Approaches

This chapter provides a detailed exposition of our methodological approach. Utilizing Coze (https://www.coze.com), we developed a MAS for EAP writing assistance, which we have termed AcademiCraft. The architectural schema of the system is depicted in Figure 1. AcademiCraft divides the task of EAP writing assistance into multiple granular subtasks via nine distinct agents. These agents collaboratively transform a user-submitted draft of EAP text into a manuscript that meets publication standards, providing comprehensive and justified revisions. The following discussion will detail the specific functions of each agent within the system.

The arrangement agent is responsible for managing and operating the entire MAS. It also plays a crucial role in interacting with users. Upon receiving a user’s EAP draft, the arrangement agent distributes the document to the GEC agent, followed by the execution of the other agents in sequence. The final output includes a revised EAP text along with justifications for the revisions made, which are then presented to the user.

The GEC agent is designed for GEC in the EAP draft. Solutions such as those proposed [42,43] integrate GEC directly into the drafting process. However, this integration of additional variables may potentially diminish the quality of either the GEC or the draft revisions. Ref. [44]’s research has demonstrated the exceptional performance of LLMs in the GEC task. Therefore, utilizing an LLM-based Agent for GEC is evidently a prudent choice. Our GEC agent is developed in accordance with [45]’s design principles to ensure the quality of GEC.

Node switch agent is designed to classify EAP texts and direct them to different nodes based on their type. Our approach categorizes the grammar-corrected EAP drafts into three distinct textual dimensions: sentence, paragraph, and chapter. This differentiation is crucial because the focus points of academic texts vary significantly depending on their structure [46,47]. For clarity, we define these textual dimensions as follows:

• Sentence refers to a single, standalone sentence that requires revision primarily focused on grammatical accuracy, academic phraseology, and lexical precision.
• Paragraph encompasses text consisting of two or more sentences forming a cohesive unit with a central idea, requiring analysis of internal coherence, logical flow, and collective academic tone.
• Chapter refers to multiple paragraphs organized together, requiring higher-level structural assessment including moves analysis and cross-paragraph coherence.

By segmenting the EAP drafts into these distinct granularity levels, our system employs a divide-and-conquer strategy, ensuring efficient processing and enhancement of academic writing. It is important to note that these three dimensions operate independently, with no direct communication between their respective agents. The node switch agent serves as a classifier that determines which specific revision path is appropriate based on the input text’s length and structure, then directs the text exclusively to either the Sentence, Paragraph, or Chapter processing track.

Each textual dimension is processed by its dedicated agents: Sentence-level texts are handled solely by the sentence revision agent and evaluated by the sentence evaluation agent; paragraph-level texts are processed exclusively by the paragraph revision agent and assessed by the paragraph evaluation agent; chapter-level texts are managed only by the chapter revision agent with evaluation from the chapter evaluation agent.

The revision process follows a single-pass approach rather than an iterative cycle. Each text is processed exactly once through its respective dimension-specific agents, with no repeated revisions or feedback loops across different dimensions. This design decision was made to maintain system stability and ensure that the final output remains recognizable to users. Our preliminary experiments indicated that multiple revision cycles could potentially lead to text that increasingly diverges from the author’s original intent and style, which might reduce user acceptance of the suggested changes.

Although the system does not implement multiple revision cycles, we have carefully designed each dimension-specific agent to consider all relevant aspects of academic writing appropriate to its level. For instance, the chapter revision agent performs comprehensive analysis by incorporating both micro-level linguistic features and macro-level organizational structures in a single pass. This approach ensures effective enhancement while preserving the author’s voice and maintaining a reasonable boundary on the extent of revisions.

The arrangement agent manages the overall workflow, ensuring that each text is properly classified, directed to the appropriate dimension-specific agents, and that the evaluation results are properly formatted for user presentation. This clearly delineated approach allows for specialized processing at each textual dimension while maintaining system modularity, which facilitates both effective performance and clear evaluation across the three distinct textual dimensions, as demonstrated in our experimental results.

The sentence revision agent is designed for sentence-level revision. For EAP writing, whether for L1 or L2 users, the core focus is to grasp the academic writing structure of sentences [48,49,50]. Central to this is the use of formulaic language (FL). Utilizing FL in sentence-level EAP writing is akin to a beginner swimmer using a flotation device—making the process considerably easier [51]. Writers can simply apply EAP sentence formulas to construct proper academic sentences.

Among the components of FL, lexical bundles (LBs) are particularly effective. This concept, first introduced by [52], refers to recurrent sequences of words that commonly appear together in academic texts and serve specific communicative functions. LBs are typically multi-word expressions that help in structuring arguments, presenting evidence, and framing research findings in a way that is conventional in academic writing.

For example, LBs can include phrases such as the following: “as shown in Figure”, “the results of the study” and “it is important to note that”. These phrases are not only common in academic texts but also help writers convey their ideas more clearly and effectively by using standardized expressions that are widely recognized and understood within the academic community.

In a previous study [53], LBs were employed in Seq2seq models for sentence-level EAP writing assistance systems. Although this model showed performance improvements over the earlier solution [54], the improvements were marginal. This limited enhancement was due to the direct substitution of unsuitable phrases with EAP LBs through N-gram matching. This approach resulted in minimal revisions to the draft sentence structure and failed to identify and replace many non-contiguous LBs.

The core idea behind our designed sentence revision agent is to match sentences with an Academic Phrasebank, ensuring that the entire sentence draft fully incorporates the LBs from the Academic Phrasebank, rather than merely performing partial structure retrieval and substitution. Recent advancements in LLMs possess extensive background knowledge, making them capable of revising EAP sentence drafts as proficiently as an expert in EAP linguistics.

The paragraph revision agent is designed to enhance paragraph quality by focusing on overall structure rather than individual sentences. In EAP writing, paragraph analysis is essential, encompassing several key aspects:

First, the main idea or argument of the paragraph must be clearly identified and presented, ensuring that the central idea is prominent and unambiguous [13]. The agent then conducts a thorough analysis of the paragraph’s structure, including the topic sentence, supporting sentences, and concluding sentence, to ensure logical progression and clarity. Improving the coherence and logical flow between sentences is critical, so the agent assesses and revises the paragraph to enhance these aspects.

Accuracy, academic tone, and clarity of expression are essential in EAP writing [55]. The agent reviews the language use, ensuring that sentences are formal and precise, utilizing structures from the Academic Phrasebank [56] to enhance academic rigor. The Academic Phrasebank is a comprehensive resource developed at the University of Manchester, providing a collection of phrases and sentences commonly used in academic writing. It helps writers construct well-structured arguments and present their ideas clearly and formally. Hedges and boosters are crucial in academic writing for presenting arguments with appropriate caution or emphasis [57,58,59,60]. The agent identifies and adjusts the use of these linguistic tools to align with the academic tone and purpose of the paragraph.

Finally, the passive voice is significant in EAP for maintaining objectivity and focusing on the action rather than the actor [61]. According to [62], the use of hedges, boosters, and even passive voice are interconnected, further highlighting the linguistic complexity of EAP and the superiority of our fine-grained approach. The agent identifies passive constructions and modifies them as necessary to suit the academic context.

Chapter revision agent is designed for comprehensive chapter-level revisions, going beyond mere paragraph adjustments. This process begins with a thorough analysis and restructuring of the entire chapter, followed by detailed paragraph revisions. Specifically, the chapter revision agent first conducts a ‘moves’ analysis [63] on the EAP draft. Moves analysis, in this context, refers to identifying the functional components of the text, such as introducing the topic, establishing a territory, and creating a research space, in line with Swales’ CARS (Create a Research Space) model [64].

The CARS model is instrumental in this process. It includes steps like identifying and analyzing the moves within the text. These moves involve creating a research space, establishing a territory, and occupying a niche. By incorporating the CARS model, the chapter revision agent ensures a structured and methodical approach to text analysis.

Upon completing the moves analysis, the chapter revision agent proceeds with paragraph analysis. However, this is not a straightforward paragraph revision; it incorporates an additional element—evidence. This element is unique to EAP chapter writing and is critical in ensuring that the overall structure and syntactical style of the chapter adhere to academic standards.

The sentence evaluation agent is designed to compare a user’s draft sentence with its revised version, providing feedback on the reasons for the revisions. Specifically, the sentence evaluation agent will produce an evaluation result that details the reasons for the revisions based on two main aspects: (1) grammar and (2) use of LBs. To ensure the structural stability of the output, we employ a few-shot learning approach, which enhances the reliability and stability of the output while also increasing its interpretability [65]. Upon completing its task, the sentence evaluation agent transmits the results back to the arrangement agent, which then delivers the final output to the user.

The paragraph evaluation agent is designed to compare users’ draft paragraphs with their revised versions, providing feedback on the reasons for revisions. Unlike the conventional sentence evaluation agent, this tool assesses and explains the revisions in two distinct dimensions: grammatical level and content level. The grammatical level encompasses four aspects: (1) grammar and (2) use of LBs, (3) use of hedges and boosters, and (4) use of passive voice. The content level focuses on paragraph analysis.

The chapter evaluation agent is designed to compare users’ chapter drafts with their revised versions, providing feedback on the reasons for revisions. Building upon the capabilities of the sentence evaluation agent, the chapter evaluation agent incorporates the evaluation and explanation of moves analysis.

For detailed settings of each agent, please refer to Appendix A.

4.1. Agent Interaction and Workflow Formalization

The multi-agent architecture of AcademiCraft can be represented as a directed workflow graph $G=(V,E)$, where vertices $V=\{v_1,v_2,\dots ,v_9\}$ represent the nine distinct agents, and edges, E, represent the information flow between agents. The arrangement agent ($v_1$) serves as both the source and sink of this graph, coordinating the overall information flow.

For a given input text, T, the processing can be formalized as a sequential transformation function:

$$\mathrm{Prompt}\mathrm{engineering}=arg\underset{P}{max}\sum _{i=1}^n{\mathbb{E}}_M\left[Q\left(O_i\right|P,C_i)\right]$$

(2)

where $f_i$ represents the transformation function of agent i. The specific agents involved in the transformation depend on the text classification performed by the node switch agent. Let $C\left(T\right)\in \{S,P,C\}$ be the classification function that maps input text to one of three categories: sentence (S), paragraph (P), or chapter (C).

The transformation path can then be defined as follows:

$$f\left(T\right)=\left\{\begin{array}{cc}{f}_{\mathrm{SE}}\circ f_{\mathrm{SR}}\circ f_{\mathrm{GEC}}\left(T\right),\hfill & \mathrm{if}C\left(T\right)=S\hfill \\ f_{\mathrm{PE}}\circ f_{\mathrm{PR}}\circ f_{\mathrm{GEC}}\left(T\right),\hfill & \mathrm{if}C\left(T\right)=P\hfill \\ f_{\mathrm{CE}}\circ f_{\mathrm{CR}}\circ f_{\mathrm{GEC}}\left(T\right),\hfill & \mathrm{if}C\left(T\right)=C\hfill \end{array}\right.$$

(3)

where $f_{\mathrm{GEC}}$ represents the grammar error correction agent; $f_{\mathrm{SR}}$, $f_{\mathrm{PR}}$, and $f_{\mathrm{CR}}$ represent the sentence, paragraph, and chapter revision agents respectively; and $f_{\mathrm{SE}}$, $f_{\mathrm{PE}}$, and $f_{\mathrm{CE}}$ represent the corresponding evaluation agents.

4.2. Computational Analysis

The computational approach of AcademiCraft is influenced by the sequential nature of agent operations. For a text with n tokens, the processing can be conceptually divided into three main phases:

$$T\left(n\right)=T_{\mathrm{GEC}}\left(n\right)+T_{\mathrm{Rev}}\left(n\right)+T_{\mathrm{Eval}}\left(n\right)$$

(4)

where $T_{\mathrm{GEC}}\left(n\right)$ represents the grammar correction phase, $T_{\mathrm{Rev}}\left(n\right)$ represents the revision process, and $T_{\mathrm{Eval}}\left(n\right)$ represents the evaluation phase.

The complexity varies across different textual dimensions. Sentence-level processing involves primarily local operations. Paragraph-level processing requires analyzing relationships between sentences, incorporating coherence measures and discourse structure. Chapter-level processing encompasses more global features, including moves analysis and cross-paragraph coherence assessment.

Each agent contributes specialized processing capabilities to the overall system, with carefully designed prompts that target specific aspects of academic writing. This modular design allows for focused enhancement at each textual level while maintaining an integrated approach to writing assistance that spans from local grammatical concerns to global organizational structures.

5. Experiments

This section presents a comprehensive evaluation of AcademiCraft’s performance through a series of systematic experiments. Our evaluation framework encompasses three key dimensions: text granularity (sentence, paragraph, and chapter levels), writer proficiency (L1 and L2 English writers), and multiple performance metrics (including grammatical accuracy, textual coherence, and academic language use). We first introduce our baseline systems and evaluation datasets, followed by detailed descriptions of our evaluation metrics. Subsequently, we present quantitative results comparing AcademiCraft with leading commercial writing assistance systems across various aspects. We conclude this section with an in-depth analysis and discussion of the experimental findings, highlighting both the strengths and potential areas for improvement in our approach. This experimental evaluation aims to answer several critical questions:

• How effective is AcademiCraft at improving text quality across different granularities compared to existing commercial tools?
• Does AcademiCraft provide consistent performance benefits for both L1 and L2 English writers?
• What are the specific strengths and limitations of our MAS-based approach in different aspects of academic writing assistance systems?

5.1. Baselines

We selected several mainstream commercial EAP writing assistance products currently available on the market, as introduced in the related work, to serve as baselines. To better illustrate their characteristics, we listed several key features in

Table 1. Comparison of product features of mainstream commercial EAP writing assistance tools.

Product	Pitaya Pro	Wordvice AI Premium	QuillBot Premium	Effidit	DeepL Write Pro
Charge?	Yes	Yes	Yes	No	Yes
Function	Rewrite	AI Paraphraser	Paraphrasing Tool	Text Polishing	-
Text Limit	6000 words	10,000 words	80,000 words	300 words	None

Note: For Wordvice AI, both free and premium versions provide access to AI Paraphraser with academic definitions, but the free version has a word limit of 500. For Pitaya and QuillBot, the free version lacks access to this feature. For DeepL Write, the primary distinctions between the free and Pro versions lie in usage limits and text count restrictions.

.

5.2. Data

To evaluate the performance of different products, we selected various test data across three textual dimensions: sentences, paragraphs, and complete texts. Specifically, for sentence revision, we utilized the LOCNESS [66] and TCNAEC [67] corpora. LOCNESS is a corpus of native English essays consisting of A-level essays from British pupils, essays from British university students, and essays from American university students. In contrast, TCNAEC comprises draft sentences in academic English written by non-native English speakers from ten different countries. We randomly selected several sentences according to method x.

For paragraph and full-text revisions, we used the LOCNESS and ICNALE [68] corpora. ICNALE includes over 10,000 topic-based speeches and essays written by university students from ten non-English-speaking countries. We focused exclusively on the essay portion of ICNALE and selected several test data points using the same method.

5.3. Metrics

5.3.1. Sentence Revision

• Grammatical error rate (GER) is used to evaluate the grammatical error rate within a single text. GER measures the ratio of the number of grammatical errors to the total number of words in the text, thereby assessing the grammatical accuracy of the text. This metric is calculated using the language-tool-python (https://github.com/jxmorris12/language_tool_python (accessed on 1 July 2024)). The formula for the calculation is as follows:

  $$\mathrm{GER}={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{Grammatical}\mathrm{Errors}}{\mathrm{Total}\mathrm{Number}\mathrm{of}\mathrm{Words}}}$$

  (5)
• The proportion of grammatical error types is used to evaluate the relative frequency of various types of grammatical errors in a text. This metric reflects the distribution of different types of grammatical errors by counting the instances of each type of error and calculating their proportion in the total number of grammatical errors. This metric is calculated using the language-tool-python
• GLUE [69] is used to evaluate the semantic similarity between the draft and the revised draft. This metric employs the all-MiniLM-L6-v2 [70] model from the SentenceTransformers [71] library. Specifically, the draft and revised draft texts are encoded into vectors using the model, and then the cosine similarity between these vectors is calculated. The formula is as follows:

  $$\mathrm{GLUE}=cos({\mathbf{v}}_{\mathrm{draft}},{\mathbf{v}}_{\mathrm{revised}})$$

  (6)

  where ${\mathbf{v}}_{\mathrm{draft}}$ and ${\mathbf{v}}_{\mathrm{revised}}$ represent the vector representations of the draft and revised draft, respectively.
• Flesch reading ease (FRE) [72] is used to evaluate the readability of a text. This metric is calculated using the NLTK (https://github.com/nltk/nltk (accessed on 1 July 2024). The calculation formula is as follows:

  $$\mathrm{FRE}=206.835-1.015\left({\displaystyle \frac{\mathrm{Total}\mathrm{Words}}{\mathrm{Sentences}}}\right)-84.6\left({\displaystyle \frac{\mathrm{Syllables}}{\mathrm{Words}}}\right)$$

  (7)
  The FRE score ranges from 0 to 100, with higher values indicating easier readability. The specific segments are interpreted as follows:

  –

  0–30: Extremely difficult text, suitable for highly specialized academic articles.

  –

  31–50: Very difficult text, suitable for academic papers.

  –

  51–60: Moderately difficult text, suitable for articles at the undergraduate level and above.

  –

  61–70: Fairly easy-to-read text, suitable for high school-level articles.

  –

  71–80: Easy text, suitable for middle school-level articles.

  –

  81–100: Very easy text, suitable for elementary school-level articles.

• The Gunning fog index (GFI) [73] is used to measure the readability of a text, particularly evaluating the difficulty level for readers to comprehend the material. This metric determines the text’s complexity by calculating the proportion of sentence length and complex words. A higher value indicates a more difficult text to read. Thus, in academic writing, a high GFI value typically signifies more in-depth and specialized content. This metric is calculated using the NLTK. The specific calculation formula is as follows:

  $$\mathrm{GFI}=0.4\left({\displaystyle \frac{\mathrm{Total}\mathrm{Words}}{\mathrm{Total}\mathrm{Sentences}}}+100\times {\displaystyle \frac{\mathrm{Complex}\mathrm{Words}}{\mathrm{Total}\mathrm{Words}}}\right)$$

  (8)

5.3.2. Paragraph Revision

In addition to the evaluation metrics used for sentence revision, paragraph revision employs a broader range of metrics to better match the characteristics of paragraph texts. The following sections will provide a detailed explanation of these metrics.

• TL Freq FW Log (TFFL) is used to evaluate the frequency of academic vocabulary in the text. This metric reflects the academic level of the text by calculating the logarithmic frequency of specific frequent words (such as academic words). The calculation of this metric is based on the TAALES [74] tool, which extracts the frequency of academic vocabulary in the text and takes the logarithm to better quantify the use of academic vocabulary. The specific calculation formula is as follows:

  $$\mathrm{TFFL}=log\left({\displaystyle \frac{{\sum }_{i=1}^n{\mathrm{Freq}}_i}{n}}\right)$$

  (9)
• BNC-written bigram proportion (BWBP) is used to evaluate the use of bigrams in the text. This metric reflects the natural fluency and linguistic authenticity of the text by calculating the proportion of bigrams found in the written section of the BNC. The calculation of this metric is based on extracting bigrams from the text and comparing them with those in the BNC-written corpus to determine their usage proportion. This metric is calculated using the TAALES. The specific calculation formula is as follows:

  $$\mathrm{BWBP}={\displaystyle \frac{{\sum }_{i=1}^n\mathbb{I}({\mathrm{Bigram}}_i\in \mathrm{BNC})}{n}}$$

  (10)
• BNC-written trigram proportion (BWTP) is used to evaluate the use of trigrams in the text. This metric reflects the natural fluency and linguistic authenticity of the text by calculating the proportion of trigrams found in the written section of the BNC. This metric is calculated using the TAALES. The specific calculation formula is as follows:

  $$\mathrm{BWTP}={\displaystyle \frac{{\sum }_{i=1}^n\mathbb{I}({\mathrm{Trigram}}_i\in \mathrm{BNC})}{n}}$$

  (11)
• Syntactic overlap of sentence noun phrases (SOSNPs) is used to evaluate the repetition degree of noun phrases between adjacent sentences within a single draft, measuring the coherence of the text. By calculating the frequency and overlap of noun phrases in adjacent sentences, this metric reflects whether the author maintains consistent themes and coherent discourse between sentences. This metric is calculated using the TAACO [75]. The specific calculation formula is as follows:

  $$\mathrm{SOSNP}={\displaystyle \frac{{\sum }_{i=1}^{n-1}\mathrm{Count}(NounPhras{e}_i\cap NounPhras{e}_{i+1})}{{\sum }_{i=1}^{n-1}\mathrm{Count}\left(NounPhras{e}_i\right)}}$$

  (12)
• Syntactic overlap of sentence verb phrases (SOSVPs) is used to evaluate the repetition degree of verb phrases between adjacent sentences within a single draft, measuring the coherence of the text. By calculating the frequency and overlap of verb phrases in adjacent sentences, this metric reflects whether the author maintains consistent action or state descriptions between sentences, thereby enhancing the coherence of the text. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{SOSVP}={\displaystyle \frac{{\sum }_{i=1}^{n-1}\mathrm{Count}(VerbPhras{e}_i\cap VerbPhras{e}_{i+1})}{{\sum }_{i=1}^{n-1}\mathrm{Count}\left(VerbPhras{e}_i\right)}}$$

  (13)
• Latent semantic analysis full sentence similarity (LSAFSS) is used to evaluate the semantic similarity between sentences within a single draft. This metric measures the degree of similarity in a semantic space using latent semantic analysis (LSA) [76], thereby reflecting the coherence and consistency of the text. Specifically, LSAFSS maps each sentence into a semantic space and calculates the cosine similarity between adjacent sentences in this space. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{LSAFSS}={\displaystyle \frac{{\sum }_{i=1}^{n-1}cos(\overrightarrow{S_i},\overrightarrow{S_{i+1}})}{n-1}}$$

  (14)

  where $\overrightarrow{S_i}$ represents the vector representation of the i-th sentence in the semantic space, and n is the total number of sentences in the draft.
• Latent Dirichlet allocation full sentence similarity (LDAFSS) is used to evaluate the topic similarity between sentences within a single draft. This metric measures the degree of similarity in a topic space using latent Dirichlet allocation (LDA) [77], thereby reflecting the coherence and consistency of the text. Specifically, LDAFSS maps each sentence into a topic space and calculates the cosine similarity between adjacent sentences in this space. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{LDAFSS}={\displaystyle \frac{{\sum }_{i=1}^{n-1}cos(\overrightarrow{T_i},\overrightarrow{T_{i+1}})}{n-1}}$$

  (15)

  where $\overrightarrow{T_i}$ represents the vector representation of the i-th sentence in the topic space, and n is the total number of sentences in the draft.
• Word2Vec Full sentence similarity (W2VFSS) is used to evaluate the word vector similarity between sentences within a single draft. This metric measures the degree of similarity in a word vector space using the Word2Vec [78], thereby reflecting the coherence and consistency of the text. Specifically, W2VFSS maps each sentence into the word vector space and calculates the cosine similarity between adjacent sentences in this space. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{W}2\mathrm{VFSS}={\displaystyle \frac{{\sum }_{i=1}^{n-1}cos(\overrightarrow{W_i},\overrightarrow{W_{i+1}})}{n-1}}$$

  (16)

  where $\overrightarrow{W_i}$ represents the vector representation of the i-th sentence in the word vector space, and n is the total number of sentences in the draft.

5.3.3. Chapter Revision

• Syntactic overlap of paragraph noun phrases (SOPNPs) is used to evaluate the repetition degree of noun phrases within a single draft, measuring the internal consistency and coherence of the text. By calculating the frequency and overlap of noun phrases within the text, this metric reflects whether the author maintains a high level of consistency and logical coherence within paragraphs. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{SOPNP}={\displaystyle \frac{{\sum }_{i=1}^n\mathrm{Count}(NounPhras{e}_i\cap NounPhras{e}_{i+1})}{{\sum }_{i=1}^n\mathrm{Count}\left(NounPhras{e}_i\right)}}$$

  (17)
• Syntactic overlap of paragraph verb phrases (SOPVPs) is used to evaluate the repetition degree of verb phrases within a single draft, measuring the internal consistency and coherence of the text. By calculating the frequency and overlap of verb phrases within the text, this metric reflects whether the author maintains consistent actions or states descriptions within paragraphs, thereby enhancing the coherence of the text. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{SOPVP}={\displaystyle \frac{{\sum }_{i=1}^n\mathrm{Count}(VerbPhras{e}_i\cap VerbPhras{e}_{i+1})}{{\sum }_{i=1}^n\mathrm{Count}\left(VerbPhras{e}_i\right)}}$$

  (18)
• Latent semantic analysis full paragraph similarity (LSAFPS) is used to evaluate the semantic similarity within paragraphs of a single draft. LSAFPS measures the overall coherence and consistency of the text by comparing the semantic similarity between every pair of sentences within a paragraph. This metric is crucial for understanding the thematic coherence within paragraphs and its contribution to the overall structure of the text. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{LSAFPS}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}cos(\overrightarrow{d_i},\overrightarrow{d_j})$$

  (19)
• Latent Dirichlet allocation full paragraph similarity (LDAFPS) is used to evaluate the thematic similarity within paragraphs of a single draft. LDAFPS measures the overall coherence and consistency of the text by comparing the topic distribution between every pair of sentences within a paragraph. This metric is crucial for understanding the thematic consistency within paragraphs and its contribution to the overall structure of the text. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{LDAFPS}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathrm{JS}(\overrightarrow{t_i},\overrightarrow{t_j})$$

  (20)
• Word2Vec full paragraph similarity (W2VFPS) is used to evaluate the similarity within paragraphs of a single draft based on word vector representations. W2VFPS measures the overall coherence and consistency of the text by comparing the average word vectors between every pair of sentences within a paragraph. This metric is crucial for understanding the semantic consistency within paragraphs and its contribution to the overall structure of the text. This metric is calculated using the TAACO. The specific calculation formula is as follows:

  $$\mathrm{W}2\mathrm{VFPS}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}cos(\overrightarrow{w_i},\overrightarrow{w_j})$$

  (21)

5.4. Discussion on Metrics

We opted not to employ LLMs-based methods for evaluation. Despite some studies highlighting the considerable potential of these methods over traditional approaches [79,80,81], they are fraught with uncertainties [82,83]. Our objective in this work is to minimize uncertainties as much as possible during the evaluation process. Additionally, metrics such as BLEU [84] and Perplexity [85] have been widely used; however, the studies by [86,87] have revealed that these metrics do not align well with human evaluation characteristics, leading us to exclude them from our evaluation framework.

6. Results

Based on our scientific experiments and evaluations, we obtained notable results from our benchmark assessments. It is worth highlighting that Effidit consistently underperformed across all tests. Specifically, in sentence and paragraph revision tasks, Effidit’s output was highly inconsistent. For the same sentence or paragraph input, Effidit would occasionally indicate that the input was perfect, sometimes provide a revised version, and at other times, inexplicably translate the input into Chinese. This inconsistency suggests that Effidit’s system architecture may incorporate a pre-trained machine translation model. Furthermore, for passage-level revisions, Effidit’s limitations on the number of input attempts made testing infeasible. Despite being freely available, Effidit, as a commercial product, proves to be disappointing.

In contrast, the other tools demonstrated their respective strengths in various aspects. The following sections will present the results in detail and provide a thorough analysis.

6.1. Examples

We will first present the performance of our tool, AcademiCraft. Figure 2 illustrates an example of sentence revision using AcademiCraft Due to text length constraints, examples of paragraph and chapter revisions are provided in Appendix B in textual form.

6.2. GEC Evaluation Results

6.2.1. GER Score

For the GER scores of different tools, please refer to

Table 2. Evaluation results of GER.

Text Level	Corpus	Draft	Pitaya	QuillBot	DeepL Write	Wordvice AI	AcademiCraft
			Pro	Premium	Pro	Premium	(Our)
Sentence	LOCNESS	2.90	0.50	0.44	0.91	0.50	0
	TCNAEC	1.65	1.67	1.35	3.65	1.21	1.53
Paragraph	LOCNESS	2.26	0.43	0.08	0.65	0.02	0.06
	ICNALE	1.04	0.06	0.09	0.52	0.06	0.14
Chapter	LOCNESS	3.43	1.36	1.12	1.31	0.12	0.90
	ICNALE	3.89	0.45	0.60	0.79	0	0

.

Analysis

Sentence-level analysis: In the L1 EAP corpus LOCNESS, AcademiCraft achieved a GER of 0, demonstrating complete mastery over L1 EAP texts. For the L2 EAP corpus TCNAEC, the GER was 1.53, slightly higher than mainstream tools such as QuillBot Premium and DeepL Write Pro, but superior to Draft and Pitaya Pro. This indicates AcademiCraft’s effective correction capabilities in L2 EAP writing, albeit with room for improvement.

Paragraph-level analysis: AcademiCraft exhibited a GER of 0.06 in the LOCNESS corpus, showcasing its ability to maintain grammatical accuracy at the paragraph level. In the ICNALE corpus, the GER increased to 0.14, which, while higher than that of DeepL Write Pro, was lower than other competing tools, confirming its efficacy in processing L2 paragraphs.

Chapter level analysis: In the LOCNESS corpus, AcademiCraft recorded a GER of 0.90, outperforming other tools and demonstrating strong capability in maintaining grammatical accuracy across extensive texts. In the ICNALE corpus, a GER of 0 was achieved, indicating that AcademiCraft is highly effective in correcting complex structures in L2 EAP texts.

Comprehensive evaluation: AcademiCraft demonstrated low GER across various text levels and different corpora, particularly excelling in sentence and chapter levels for both L1 and L2 EAP texts. These findings reveal AcademiCraft’s potential in grammatical correction, especially in its handling of long texts and L2 EAP texts. Further algorithmic enhancements, particularly for L2 EAP writing as observed in the TCNAEC corpus, could improve its capacity to process complex and diverse texts.

6.2.2. The Proportion of Grammatical Error Types

Analysis

Sentence revisions in the LOCNESS Corpus: The radar charts illustrate the distribution of different grammatical error types corrected by various tools in sentence revisions within the LOCNESS corpus. As shown in Figure 3, the detailed breakdown reveals nuanced differences in error correction capabilities. AcademiCraft shows no errors, indicating its high efficiency in sentence-level corrections for L1 EAP texts. Tools like Draft and Pitaya Pro predominantly corrected errors related to possessive apostrophes and morphological rules, while Wordvice AI Premium focused solely on possessive apostrophes. These findings highlight the specialization of each tool in addressing specific error types.

Sentence revisions in the TCNAEC corpus: As shown in Figure 4, in the TCNAEC corpus, AcademiCraft demonstrates a balanced correction of morphological rules and infinitive errors. Tools like Draft and Pitaya Pro also corrected similar error types, with Pitaya Pro notably addressing a higher proportion of morphological errors. This suggests that while these tools are effective in correcting specific grammatical errors in L2 EAP texts, AcademiCraft maintains a more comprehensive approach.

Paragraph revisions in the LOCNESS corpus: For paragraph-level revisions in the LOCNESS corpus, As shown in Figure 5, AcademiCraft addresses a range of error types, including specific cases and compound sentences. Draft and DeepL Write Pro show varied distributions, focusing on different grammatical aspects like uppercase sentence starts and specific case rules. Wordvice AI Premium again focuses solely on morphological rules. The diverse error type corrections by AcademiCraft indicate its robustness in handling paragraph-level grammatical accuracy.

Paragraph revisions in the ICNALE corpus: As shown in Figure 6, in the ICNALE corpus, AcademiCraft and other tools continue to exhibit diverse correction capabilities. AcademiCraft balances corrections among morphological rules and specific grammatical constructs, whereas other tools like QuillBot Premium and Wordvice AI Premium primarily focus on a single error type, morphological rules. This consistency in correction diversity further supports AcademiCraft’s adaptability to different L2 EAP texts.

Chapter revisions in the LOCNESS corpus: For chapter-level revisions within the LOCNESS corpus, As shown in Figure 7, AcademiCraft again shows no errors, indicating exceptional performance in maintaining grammatical accuracy across longer texts. Other tools, such as Draft and Pitaya Pro, focus on possessive apostrophes and compound sentences. The significant error-free performance of AcademiCraft underscores its effectiveness in comprehensive text revisions.

Chapter revisions in the ICNALE corpus: As shown in Figure 8, in the ICNALE corpus, both AcademiCraft and Wordvice AI Premium show no errors, suggesting a high level of efficiency in managing grammatical accuracy for L2 EAP chapter revisions. Tools like Draft and Pitaya Pro show varied distributions, addressing multiple error types. This consistency of error-free results by AcademiCraft highlights its reliability across extensive texts.

Comprehensive evaluation: The analysis across different text levels and corpora demonstrates AcademiCraft’s superior performance in grammatical corrections. Its ability to address a wide range of grammatical errors, especially in L2 EAP texts, sets it apart from other tools that tend to specialize in specific error types. The tool’s consistent performance across sentences, paragraphs, and chapters indicates its robustness and adaptability, making it a highly effective solution for enhancing grammatical accuracy in academic writing. Further improvements could focus on maintaining this high level of performance across even more diverse and complex textual structures.

6.3. Evaluation Results of Statistical Metrics

Analysis

As detailed in

Table 3. Evaluation results of statistical metrics.

Text Level	Corpus	Text	TFFL	BWBP	BWTP	SOSNP	SOSVP	SOPNP	SOPVP
Paragraph	LOCNESS	Draft	4.66	0.54	0.19	0.82	0.76	-	-
		Pitaya Pro	4.73	0.42	0.13	0.80	0.25	-	-
		QuillBot Premium	4.72	0.47	0.16	0.83	0.34	-	-
		DeepL Write Pro	4.77	0.53	0.22	0.56	0.46	-	-
		Wordvice AI Premium	4.71	0.44	0.15	1.05	0.34	-	-
		AcademiCraft (Our)	4.66	0.43	0.14	0.82	0.26	-	-
	ICNALE	Draft	4.57	0.49	0.15	0.92	0.59	-	-
		Pitaya Pro	4.68	0.34	0.07	0.84	0.25	-	-
		QuillBot Premium	4.62	0.38	0.08	0.78	0.33	-	-
		DeepL Write Pro	4.75	0.45	0.09	1.22	0.45	-	-
		Wordvice AI Premium	4.67	0.36	0.14	1.17	0.24	-	-
		AcademiCraft (Our)	4.65	0.35	0.06	0.92	0.18	-	-
Chapter	LOCNESS	Draft	4.68	0.41	0.16	0.89	0.58	4.65	5.03
		Pitaya Pro	2.57	0.33	0.09	0.80	0.28	5.93	4.19
		QuillBot Premium	4.73	0.37	0.12	0.83	0.39	6.97	6.99
		DeepL Write Pro	4.79	0.40	0.16	0.80	0.41	8.78	6.71
		Wordvice AI Premium	4.75	0.35	0.10	1.12	0.53	6.31	2.90
		AcademiCraft (Our)	4.69	0.37	0.11	0.95	0.41	7.59	5.47
	ICNALE	Draft	4.57	0.47	0.14	1.08	0.89	5.45	4.71
		Pitaya Pro	4.64	0.36	0.07	0.61	0.23	4.31	1.23
		QuillBot Premium	4.67	0.38	0.10	0.89	0.54	5.20	2.51
		DeepL Write Pro	4.74	0.44	0.14	0.82	0.33	4.34	2.44
		Wordvice AI Premium	4.64	0.33	0.06	1.14	0.16	4.03	0.50
		AcademiCraft (Our)	4.59	0.36	0.08	0.74	0.24	6.49	2.85

, the evaluation results provide a comprehensive overview of the statistical metrics across different writing assistance systems and corpora.

Paragraph-level analysis: In the LOCNESS corpus, TFFL values indicate that the writing assistance systems performed similarly, ranging from 4.66 to 4.77. Draft and AcademiCraft scored slightly lower (4.66), while DeepL Write Pro scored the highest (4.77). However, a higher TFFL does not necessarily indicate better text quality, as it may merely reflect changes in word frequency distribution [88].

BWBP and BWTP measure the proportion of bigrams and trigrams, respectively. Draft has a notably higher BWBP (0.54) than the other tools, while AcademiCraft (0.43) and DeepL Write Pro (0.53) have relatively lower scores. This suggests that Draft may contain more common phrase combinations, which does not inherently imply better language quality. Conversely, lower BWBP and BWTP values might indicate a richer vocabulary use [89].

For syntactic overlap metrics (SOSNP, SOSVP), DeepL Write Pro shows higher values in SOSNP (0.56) and SOSVP (0.46), indicating more consistent syntactic structures in its revisions. In contrast, AcademiCraft’s performance in SOSNP (0.82) and SOSVP (0.26) is relatively lower, potentially indicating more varied syntactic structures post-revision and slightly weaker syntactic consistency.

In the ICNALE corpus, the TFFL distribution is similar to that of LOCNESS, with DeepL Write Pro (4.75) again performing best, and AcademiCraft (4.65) close behind. Higher TFFL values might indicate a more concentrated word frequency distribution for L2 learners, but this does not necessarily correlate with higher text quality. For syntactic overlap metrics (SOSNP, SOSVP), DeepL Write Pro once more demonstrates higher scores (1.22 and 0.45), suggesting higher consistency in L2 corpus syntactic structures, while AcademiCraft performs moderately (0.92 and 0.18).

Chapter-level analysis: In the LOCNESS corpus, TFFL values range from 4.68 to 4.79, with Draft (4.68) and AcademiCraft (4.69) performing slightly lower and DeepL Write Pro (4.79) the highest. The BWBP and BWTP distribution shows Draft scoring higher in BWBP (0.41), indicating more common bigram usage, but this does not necessarily signify quality. AcademiCraft’s scores in BWBP (0.37) and BWTP (0.11) are moderate.

Syntactic overlap metrics (SOPNP, SOPVP) indicate that DeepL Write Pro excels in SOPNP (8.78) and SOPVP (6.71), showing high consistency in paragraph-level syntactic structures. AcademiCraft also performs well in SOPNP (7.59) and SOPVP (5.47), indicating good consistency in paragraph-level syntax, though slightly inferior to DeepL Write Pro.

In the ICNALE corpus, TFFL values are concentrated, with Draft (4.57) and AcademiCraft (4.59) on the lower end and DeepL Write Pro (4.74) leading. The BWBP and BWTP distribution is similar to the Paragraph level, with Draft performing higher in BWBP (0.47) and AcademiCraft (0.36) being moderate. For syntactic overlap metrics, AcademiCraft shows good performance in SOPNP (6.49) and SOPVP (2.85), although it is still slightly behind DeepL Write Pro (4.34 and 2.44).

Comprehensive evaluation: Overall, Draft shows higher values in several metrics, but these do not necessarily indicate better text quality, as higher word frequency features and bigram proportions may merely reflect common vocabulary usage rather than superior writing. AcademiCraft performs well across multiple metrics, especially in TFFL and syntactic overlap, although it has room for improvement in some areas. Its overall stable performance indicates it is a competitive EAP writing assistance system.

In comparing L1 (LOCNESS) and L2 (ICNALE) corpora, it is evident that L2 learners’ texts face more challenges in vocabulary frequency and syntactic consistency [90]. AcademiCraft shows good revision effects in both corpora, particularly in syntactic overlap and paragraph-level consistency, indicating its potential as an EAP writing assistance system.

While AcademiCraft is slightly behind DeepL Write Pro in certain metrics, its overall performance meets the needs of EAP writing, especially in balancing vocabulary richness and syntactic consistency [8]. Future improvements can focus on enhancing syntactic consistency and improving the revision effects for L2 learners to further enhance the overall quality and effectiveness of the writing assistance system.

6.4. Evaluation Results of Performance Metrics

Analysis

As shown in

Table 4. Evaluation results of performance metrics.

Text Level	Corpus	Text	FRE	GFI	GLUE	LSAFSS	LDAFSS	W2VFSS	LSAFPS	LDAFPS	W2VFPS
Sentence	LOCNESS	Draft	52.68	13.87	-	-	-	-	-	-	-
		Pitaya Pro	33.22	17.27	0.81	-	-	-	-	-	-
		QuillBot Premium	38.76	16.27	0.86	-	-	-	-	-	-
		DeepL Write Pro	28.69	18.94	0.82	-	-	-	-	-	-
		Wordvice AI Premium	32.88	17.32	0.81	-	-	-	-	-	-
		AcademiCraft (Our)	13.10	21.60	0.74	-	-	-	-	-	-
	TCNAEC	Draft	52.66	13.30	-	-	-	-	-	-	-
		Pitaya Pro	22.47	18.48	0.85	-	-	-	-	-	-
		QuillBot Premium	34.27	16.85	0.91	-	-	-	-	-	-
		DeepL Write Pro	25.78	19.38	0.88	-	-	-	-	-	-
		Wordvice AI Premium	10.55	21.83	0.81	-	-	-	-	-	-
		AcademiCraft (Our)	11.37	22.09	0.84	-	-	-	-	-	-
Paragraph	LOCNESS	Draft	53.43	14.84	-	0.47	0.95	0.79	-	-	-
		Pitaya Pro	36.98	17.71	0.89	0.46	0.93	0.78	-	-	-
		QuillBot Premium	35.22	17.74	0.89	0.47	0.91	0.81	-	-	-
		DeepL Write Pro	36.20	18.14	0.89	0.49	0.94	0.82	-	-	-
		Wordvice AI Premium	24.80	20.34	0.85	0.49	0.87	0.78	-	-	-
		AcademiCraft (Our)	35.83	17.66	0.90	0.51	0.88	0.80	-	-	-
	ICNALE	Draft	42.08	17.87	-	0.43	0.84	0.69	-	-	-
		Pitaya Pro	30.40	17.70	0.86	0.39	0.89	0.71	-	-	-
		QuillBot Premium	27.24	18.63	0.84	0.43	0.91	0.75	-	-	-
		DeepL Write Pro	27.90	19.38	0.88	0.47	0.95	0.81	-	-	-
		Wordvice AI Premium	15.08	21.56	0.88	0.45	0.85	0.74	-	-	-
		AcademiCraft (Our)	23.24	18.74	0.89	0.50	0.92	0.80	-	-	-
Chapter	LOCNESS	Draft	57.41	13.53	-	0.49	0.97	0.87	0.57	0.99	0.89
		Pitaya Pro	37.30	16.98	0.90	0.48	0.94	0.88	0.59	0.94	0.90
		QuillBot Premium	34.59	17.77	0.90	0.49	0.95	0.89	0.48	0.79	0.73
		DeepL Write Pro	38.53	16.89	0.89	0.50	0.95	0.89	0.64	0.97	0.91
		Wordvice AI Premium	23.62	20.72	0.84	0.55	0.97	0.91	0.59	0.84	0.78
		AcademiCraft (Our)	41.48	16.35	0.90	0.50	0.96	0.89	0.64	0.97	0.91
	ICNALE	Draft	58.96	13.39	-	0.58	0.97	0.88	0.43	0.58	0.53
		Pitaya Pro	34.74	16.06	0.87	0.53	0.95	0.89	0.42	0.56	0.52
		QuillBot Premium	29.37	17.86	0.82	0.55	0.95	0.90	0.44	0.57	0.52
		DeepL Write Pro	31.86	18.03	0.80	0.58	0.96	0.90	0.47	0.54	0.56
		Wordvice AI Premium	18.22	19.73	0.78	0.64	0.97	0.91	0.35	0.43	0.40
		AcademiCraft (Our)	33.32	16.79	0.86	0.62	0.98	0.91	0.65	0.82	0.76

Bold entries indicate the best performance within each category across different tools.

, the detailed performance metrics provide insights into the various writing assistance tools’ capabilities.

Sentence level: In the sentence level evaluation of the LOCNESS corpus, AcademiCraft demonstrated a significant reduction in text complexity, evidenced by its FRE score of 13.60 and GFI score of 21.60. The GLUE score of 0.74, while slightly lower than other tools (e.g., QuillBot Premium at 0.86 and DeepL Write Pro at 0.82), still maintained a high level of semantic similarity. Compared to other tools, AcademiCraft achieved a favorable balance between reducing text complexity and maintaining semantic similarity [91,92], which is crucial in EAP writing revisions.

For the TCNAC corpus, AcademiCraft again showcased its strength in controlling text complexity with a GFI of 22.09 and FRE of 11.37. The GLUE score of 0.84, comparable to other tools (e.g., QuillBot Premium at 0.91 and DeepL Write Pro at 0.88), indicated excellent performance in semantic consistency. This performance is particularly relevant for both L1 and L2 students, as it helps maintain the original semantic integrity while simplifying the text [93,94].

Paragraph level: At the paragraph level, AcademiCraft’s performance in the LOCNESS corpus highlighted its capability to enhance text readability, with an FRE of 35.83 and GFI of 17.66, outperforming Wordvice AI Premium and DeepL Write Pro. The GLUE score of 0.90 further confirmed its superiority in semantic consistency. Additionally, the LSAFSS and W2VFSS scores of 0.51 and 0.88, respectively, indicated stable performance in sentence-level and paragraph-level semantic similarity. These results are particularly important for L1 students who often require more assistance in simplifying and understanding complex academic texts.

In the ICNALE corpus, AcademiCraft continued to excel with an FRE of 25.03 and GFI of 18.74, demonstrating effectiveness in handling complex texts. The GLUE score of 0.90 was on par with other tools (e.g., QuillBot Premium at 0.91). The LSAFSS and W2VFSS scores of 0.50 and 0.82, respectively, showed consistent performance in paragraph-level semantic similarity. This is crucial for L2 students who need to ensure text simplification and semantic consistency while learning and using academic English [95].

Chapter level: At the chapter level for the LOCNESS corpus, AcademiCraft exhibited outstanding performance with an FRE of 41.48 and GFI of 16.35, highlighting its significant advantage in enhancing text readability. The GLUE score of 0.90 indicated excellent performance in semantic consistency. Scores for LSAFPS and W2VFSS were 0.55 and 0.91, respectively, further affirming its superior ability to maintain semantic consistency across paragraphs. These metrics are particularly vital for L1 students dealing with extensive academic texts, aiding in better understanding and grasping complex content.

In the ICNALE corpus, AcademiCraft continued to demonstrate its advantages with an FRE of 33.32 and GFI of 16.79, underscoring its effectiveness in managing complex academic texts. The GLUE score of 0.86 confirmed its reliability in semantic consistency. The LSAFPS and W2VFSS scores of 0.62 and 0.82, respectively, further highlighted its stable performance in paragraph-level semantic similarity. This is highly beneficial for L2 students, assisting them in maintaining academic content integrity and consistency while simplifying the text.

Comprehensive evaluation: In summary, AcademiCraft consistently exhibited exceptional capabilities in enhancing text readability and maintaining semantic consistency across different levels and corpora. Compared to other EAP writing assistance systems, AcademiCraft achieved an optimal balance between reducing text complexity and preserving semantic similarity. These results validate its effectiveness in text revision for both L1 and L2 students, further establishing AcademiCraft as a leading tool in academic writing assistance.

6.5. Discussion

This study systematically evaluated the performance of AcademiCraft across different corpora and text levels. Our findings demonstrate that AcademiCraft offers significant advantages over existing commercial tools, particularly in its comprehensive approach to academic writing assistance system and explainability of revisions. While quantitative metrics show competitive performance, the system’s true strength lies in its multi-dimensional analysis capabilities and transparent feedback mechanisms.

At the sentence level, AcademiCraft achieved a zero-error rate in the L1 EAP corpus (LOCNESS), showcasing its exceptional ability to handle academic English writing by native speakers. Although tools like QuillBot Premium and DeepL Write Pro showed marginally lower error rates in the L2 EAP corpus (TCNAEC), AcademiCraft significantly outperformed them in maintaining semantic similarity between original and revised texts, as evidenced by superior GLUE scores. This balance between correction and preservation of original meaning represents a critical advantage in academic writing contexts, where precision of expression is paramount.

The multi-dimensional analysis approach of AcademiCraft demonstrates particular strengths when handling larger text structures. In both paragraph and chapter levels, our system shows remarkably consistent performance across readability metrics (FRE and GFI) while simultaneously improving academic language markers like TFFL and bigram/trigram proportions. Unlike commercial tools that often focus primarily on sentence-level corrections, AcademiCraft maintains a holistic view of text coherence, as demonstrated by consistently higher SOSNP, LSAFSS, and W2VFSS scores across both L1 and L2 corpora.

A distinctive feature of AcademiCraft is its ability to provide explanations for suggested revisions—an innovation largely absent in commercial tools. While our current evaluation focused on automated metrics for textual improvements, we acknowledge that the educational value of these explanations requires separate human evaluation. The generated explanations vary in length and detail across different textual dimensions, which may present different challenges and benefits for L1 versus L2 users. Some explanations offer comprehensive linguistic rationales that could enhance deeper understanding, while others provide concise feedback that might be more immediately actionable but potentially less instructive for non-native writers.

The formulation of standardized evaluation criteria for explanation quality presents significant challenges, including the need to account for varying user proficiency levels, learning objectives, and disciplinary contexts. This complexity prevented us from conducting formal human evaluation of explanation quality in the current study. Future research should address this limitation by developing robust frameworks for assessing explanation clarity, relevance, educational value, and efficiency for different user groups. Such evaluation would provide valuable insights into optimizing the balance between comprehensiveness and conciseness in revision explanations.

Looking toward future improvements, we identify several promising directions. First, enhancing the performance on L2 texts represents a clear opportunity, particularly for addressing complex sentence structures common in academic writing by non-native speakers. Second, incorporating more sophisticated discourse analysis techniques could further strengthen chapter-level revisions. Third, exploring adaptive approaches that customize both revision strategies and explanation depth based on detected writer proficiency levels could improve overall effectiveness and educational impact. Finally, a systematic investigation of the pedagogical efficacy of revision explanations across different academic contexts and user groups would significantly contribute to understanding the educational potential of such systems.

In conclusion, despite comparable performance on some isolated metrics, AcademiCraft demonstrates clear advantages in its comprehensive approach to academic writing assistance system, consistent performance across textual dimensions, and innovative provision of revision explanations. While the current evaluation establishes its technical capabilities through automated metrics, realizing the full educational potential of such systems will require dedicated studies on how different user groups interact with and learn from the explanatory components. The development of AcademiCraft represents an important step toward writing assistance tools that not only improve texts but potentially enhance writers’ understanding of academic discourse conventions.

7. Conclusions

In this study, we developed and evaluated AcademiCraft, an English academic writing assistance based on a MAS, which surpassed mainstream commercial products on several key performance metrics, particularly in GEC and EAP text polishing. AcademiCraft stands out by providing detailed revision explanations, a feature not available in current commercial offerings. Its open and replicable technological framework contrasts sharply with the proprietary nature of most commercial tools, facilitating transparency and allowing researchers and developers to test, adjust, and enhance the technology. Our analyses across various text levels—sentence, paragraph, and chapter—demonstrate AcademiCraft’s robust capabilities in maintaining grammatical accuracy and textual consistency, especially in complex L2 EAP contexts. The results underscore AcademiCraft’s significant potential and superiority in the field of academic writing assistance system, and future work will focus on refining its algorithms to enhance its efficiency in processing a broader range of EAP texts, thereby reinforcing its leadership in the academic writing support tools market.